[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790394221564982-779-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:43:41 dist-test-slave-n2th krb5kdc[786](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:43:41 dist-test-slave-n2th krb5kdc[786](info): set up 2 sockets
Sep 26 03:43:41 dist-test-slave-n2th krb5kdc[786](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260926 03:43:42.704668 779 mini_kdc.cc:121] Time spent starting KDC: real 1.050s user 0.007s sys 0.009s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790394221564982-779-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790394221564982-779-0/krb5kdc/test-user.keytab.
Sep 26 03:43:42 dist-test-slave-n2th krb5kdc[786](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394222, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:43:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:43:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.194.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.194.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.194.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790394221564982-779-0/krb5kdc/kudu_127.0.194.254.keytab.
Entry for principal kudu/127.0.194.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790394221564982-779-0/krb5kdc/kudu_127.0.194.254.keytab.
I20260926 03:43:42.900748 779 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
/tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790394221564982-779-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790394221564982-779-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790394221564982-779-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790394221564982-779-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.194.254:43495
--webserver_interface=127.0.194.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790394221564982-779-0/krb5kdc/kudu_127.0.194.254.keytab
--principal=kudu/127.0.194.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.194.212:39905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.194.254:43495
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790394221564982-779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790394221564982-779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790394221564982-779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:43:43.222457 799 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:43:43.222802 799 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:43:43.222981 799 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:43:43.230219 799 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:43:43.230337 799 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:43:43.230470 799 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:43:43.230571 799 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:43:43.230655 799 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:43:43.240463 799 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.194.212:39905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790394221564982-779-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790394221564982-779-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.194.254:43495
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790394221564982-779-0/krb5kdc/kudu_127.0.194.254.keytab
--principal=kudu/127.0.194.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.194.254:43495
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790394221564982-779-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.194.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790394221564982-779-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:43:43.241835 799 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:43:43.243283 799 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:43:43.255190 807 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:43:43.255249 805 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:43:43.255277 804 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:43:43.255977 799 server_base.cc:1034] running on GCE node
I20260926 03:43:43.256812 799 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:43:43.258286 799 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:43:43.259536 799 hybrid_clock.cc:648] HybridClock initialized: now 1790394223259436 us; error 92 us; skew 500 ppm
I20260926 03:43:43.259898 799 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:43:43 dist-test-slave-n2th krb5kdc[786](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394223, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:43:43.263005 799 init.cc:377] Logged in from keytab as kudu/127.0.194.254@KRBTEST.COM (short username kudu)
I20260926 03:43:43.264498 799 webserver.cc:460] Webserver started at http://127.0.194.254:39235/ using document root <none> and password file <none>
I20260926 03:43:43.265214 799 fs_manager.cc:359] Metadata directory not provided
I20260926 03:43:43.265388 799 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:43:43.265714 799 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:43:43.268962 799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790394221564982-779-0/minicluster-data/master-0/data/instance:
uuid: "9cfcbe3cae1a4436896502a066189bbe"
format_stamp: "Formatted at 2026-09-26 03:43:43 on dist-test-slave-n2th"
server_key: "752dca20516ea0e281dd874f626bee37"
server_key_iv: "00d861f70caf53c664d2ccc4acd929c5"
server_key_version: "encryptionkey@0"
I20260926 03:43:43.269788 799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790394221564982-779-0/minicluster-data/master-0/wal/instance:
uuid: "9cfcbe3cae1a4436896502a066189bbe"
format_stamp: "Formatted at 2026-09-26 03:43:43 on dist-test-slave-n2th"
server_key: "752dca20516ea0e281dd874f626bee37"
server_key_iv: "00d861f70caf53c664d2ccc4acd929c5"
server_key_version: "encryptionkey@0"
I20260926 03:43:43.275849 799 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260926 03:43:43.279417 814 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:43:43.280786 799 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 03:43:43.280979 799 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790394221564982-779-0/minicluster-data/master-0/data,/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790394221564982-779-0/minicluster-data/master-0/wal
uuid: "9cfcbe3cae1a4436896502a066189bbe"
format_stamp: "Formatted at 2026-09-26 03:43:43 on dist-test-slave-n2th"
server_key: "752dca20516ea0e281dd874f626bee37"
server_key_iv: "00d861f70caf53c664d2ccc4acd929c5"
server_key_version: "encryptionkey@0"
I20260926 03:43:43.281186 799 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790394221564982-779-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790394221564982-779-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790394221564982-779-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:43:43.293911 799 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:43:43.298133 799 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:43:43.298462 799 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:43:43.325531 799 rpc_server.cc:307] RPC server started. Bound to: 127.0.194.254:43495
I20260926 03:43:43.325527 865 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.194.254:43495 every 8 connection(s)
I20260926 03:43:43.327163 799 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790394221564982-779-0/minicluster-data/master-0/data/info.pb
I20260926 03:43:43.330696 779 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu as pid 799
I20260926 03:43:43.331100 779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790394221564982-779-0/minicluster-data/master-0/wal/instance
I20260926 03:43:43.331166 866 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:43:43.332983 779 external_mini_cluster.cc:1428] Setting key 5f07e00a7b448ac8abf7ad654841c41d
I20260926 03:43:43.342370 866 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9cfcbe3cae1a4436896502a066189bbe: Bootstrap starting.
I20260926 03:43:43.345402 866 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9cfcbe3cae1a4436896502a066189bbe: Neither blocks nor log segments found. Creating new log.
I20260926 03:43:43.346403 866 log.cc:824] T 00000000000000000000000000000000 P 9cfcbe3cae1a4436896502a066189bbe: Log is configured to *not* fsync() on all Append() calls
I20260926 03:43:43.349459 866 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9cfcbe3cae1a4436896502a066189bbe: No bootstrap required, opened a new log
I20260926 03:43:43.356659 866 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9cfcbe3cae1a4436896502a066189bbe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cfcbe3cae1a4436896502a066189bbe" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 43495 } }
I20260926 03:43:43.357043 866 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9cfcbe3cae1a4436896502a066189bbe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:43:43.357146 866 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9cfcbe3cae1a4436896502a066189bbe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9cfcbe3cae1a4436896502a066189bbe, State: Initialized, Role: FOLLOWER
I20260926 03:43:43.357848 866 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9cfcbe3cae1a4436896502a066189bbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cfcbe3cae1a4436896502a066189bbe" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 43495 } }
I20260926 03:43:43.358151 866 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9cfcbe3cae1a4436896502a066189bbe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:43:43.358307 866 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9cfcbe3cae1a4436896502a066189bbe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:43:43.358500 866 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9cfcbe3cae1a4436896502a066189bbe [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:43:43.360785 866 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9cfcbe3cae1a4436896502a066189bbe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cfcbe3cae1a4436896502a066189bbe" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 43495 } }
Sep 26 03:43:43 dist-test-slave-n2th krb5kdc[786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394222, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.194.254@KRBTEST.COM
I20260926 03:43:43.361399 866 leader_election.cc:302] T 00000000000000000000000000000000 P 9cfcbe3cae1a4436896502a066189bbe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9cfcbe3cae1a4436896502a066189bbe; no voters:
I20260926 03:43:43.362079 866 leader_election.cc:288] T 00000000000000000000000000000000 P 9cfcbe3cae1a4436896502a066189bbe [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:43:43.362202 871 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9cfcbe3cae1a4436896502a066189bbe [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:43:43.363790 871 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9cfcbe3cae1a4436896502a066189bbe [term 1 LEADER]: Becoming Leader. State: Replica: 9cfcbe3cae1a4436896502a066189bbe, State: Running, Role: LEADER
I20260926 03:43:43.364394 871 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9cfcbe3cae1a4436896502a066189bbe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cfcbe3cae1a4436896502a066189bbe" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 43495 } }
I20260926 03:43:43.364646 866 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9cfcbe3cae1a4436896502a066189bbe [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:43:43.367241 873 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9cfcbe3cae1a4436896502a066189bbe [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9cfcbe3cae1a4436896502a066189bbe. Latest consensus state: current_term: 1 leader_uuid: "9cfcbe3cae1a4436896502a066189bbe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cfcbe3cae1a4436896502a066189bbe" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 43495 } } }
I20260926 03:43:43.367617 873 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9cfcbe3cae1a4436896502a066189bbe [sys.catalog]: This master's current role is: LEADER
I20260926 03:43:43.368201 872 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9cfcbe3cae1a4436896502a066189bbe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9cfcbe3cae1a4436896502a066189bbe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cfcbe3cae1a4436896502a066189bbe" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 43495 } } }
I20260926 03:43:43.368487 872 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9cfcbe3cae1a4436896502a066189bbe [sys.catalog]: This master's current role is: LEADER
I20260926 03:43:43.376447 876 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:43:43.377876 868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:43.336859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51440 (local address 127.0.194.254:43495)
0926 03:43:43.337676 (+ 817us) server_negotiation.cc:206] Beginning negotiation
0926 03:43:43.337712 (+ 36us) server_negotiation.cc:399] Waiting for connection header
0926 03:43:43.339022 (+ 1310us) server_negotiation.cc:407] Connection header received
0926 03:43:43.348453 (+ 9431us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:43:43.348473 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:43:43.348851 (+ 378us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:43:43.349257 (+ 406us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:43:43.352250 (+ 2993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:43.353613 (+ 1363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:43.355369 (+ 1756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:43.356064 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:43.356261 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:43:43.363040 (+ 6779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:43:43.363055 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:43:43.363074 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:43:43.363120 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:43:43.368204 (+ 5084us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:43:43.369629 (+ 1425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:43:43.369642 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:43:43.369653 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:43:43.369914 (+ 261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:43:43.370631 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:43:43.370638 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:43:43.370644 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:43:43.371555 (+ 911us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:43:43.375282 (+ 3727us) server_negotiation.cc:1035] Waiting for connection context
0926 03:43:43.375631 (+ 349us) server_negotiation.cc:299] Negotiation successful
0926 03:43:43.376195 (+ 564us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":610,"thread_start_us":368,"threads_started":1}
I20260926 03:43:43.383366 876 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:43:43.393023 876 catalog_manager.cc:1347] Generated new cluster ID: f4b3429a6a1e4a06bbb0912fa8fba99c
I20260926 03:43:43.393185 876 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:43:43.407760 876 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:43:43.409718 876 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:43:43.422227 876 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9cfcbe3cae1a4436896502a066189bbe: Generated new TSK 0
I20260926 03:43:43.423447 876 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.194.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.194.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.194.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790394221564982-779-0/krb5kdc/kudu_127.0.194.193.keytab.
Entry for principal kudu/127.0.194.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790394221564982-779-0/krb5kdc/kudu_127.0.194.193.keytab.
I20260926 03:43:43.472257 779 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
/tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790394221564982-779-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790394221564982-779-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790394221564982-779-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790394221564982-779-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.194.193:0
--local_ip_for_outbound_sockets=127.0.194.193
--webserver_interface=127.0.194.193
--webserver_port=0
--tserver_master_addrs=127.0.194.254:43495
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790394221564982-779-0/krb5kdc/kudu_127.0.194.193.keytab
--principal=kudu/127.0.194.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.194.212:39905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790394221564982-779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790394221564982-779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790394221564982-779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:43:43.798895 893 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:43:43.799188 893 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:43:43.799374 893 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:43:43.806998 893 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:43:43.807122 893 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:43:43.807304 893 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.194.193
I20260926 03:43:43.817444 893 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.194.212:39905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790394221564982-779-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790394221564982-779-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790394221564982-779-0/krb5kdc/kudu_127.0.194.193.keytab
--principal=kudu/127.0.194.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.194.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790394221564982-779-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.194.193
--webserver_port=0
--tserver_master_addrs=127.0.194.254:43495
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.194.193
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790394221564982-779-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:43:43.818889 893 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:43:43.820238 893 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:43:43.833223 898 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:43:43.833395 901 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:43:43.833422 899 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:43:43.834477 893 server_base.cc:1034] running on GCE node
I20260926 03:43:43.835199 893 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:43:43.836222 893 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:43:43.837477 893 hybrid_clock.cc:648] HybridClock initialized: now 1790394223837432 us; error 55 us; skew 500 ppm
I20260926 03:43:43.837934 893 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:43:43 dist-test-slave-n2th krb5kdc[786](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394223, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:43:43.841176 893 init.cc:377] Logged in from keytab as kudu/127.0.194.193@KRBTEST.COM (short username kudu)
I20260926 03:43:43.842809 893 webserver.cc:460] Webserver started at http://127.0.194.193:37489/ using document root <none> and password file <none>
I20260926 03:43:43.843528 893 fs_manager.cc:359] Metadata directory not provided
I20260926 03:43:43.843684 893 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:43:43.844017 893 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:43:43.847126 893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790394221564982-779-0/minicluster-data/ts-0/data/instance:
uuid: "5e1ed814ab1846f081466f92bb4b7ed7"
format_stamp: "Formatted at 2026-09-26 03:43:43 on dist-test-slave-n2th"
server_key: "fb05c86ea83c91ea7db86ba24a111f39"
server_key_iv: "4c0f04961feb8ed3f74d4d6b5adba6c6"
server_key_version: "encryptionkey@0"
I20260926 03:43:43.847947 893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790394221564982-779-0/minicluster-data/ts-0/wal/instance:
uuid: "5e1ed814ab1846f081466f92bb4b7ed7"
format_stamp: "Formatted at 2026-09-26 03:43:43 on dist-test-slave-n2th"
server_key: "fb05c86ea83c91ea7db86ba24a111f39"
server_key_iv: "4c0f04961feb8ed3f74d4d6b5adba6c6"
server_key_version: "encryptionkey@0"
I20260926 03:43:43.853498 893 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 03:43:43.857048 908 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:43:43.858412 893 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 03:43:43.858590 893 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790394221564982-779-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790394221564982-779-0/minicluster-data/ts-0/wal
uuid: "5e1ed814ab1846f081466f92bb4b7ed7"
format_stamp: "Formatted at 2026-09-26 03:43:43 on dist-test-slave-n2th"
server_key: "fb05c86ea83c91ea7db86ba24a111f39"
server_key_iv: "4c0f04961feb8ed3f74d4d6b5adba6c6"
server_key_version: "encryptionkey@0"
I20260926 03:43:43.858847 893 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790394221564982-779-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790394221564982-779-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790394221564982-779-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:43:43.872499 893 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:43:43.876605 893 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:43:43.876909 893 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:43:43.877847 893 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:43:43.879204 893 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:43:43.879307 893 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:43:43.879452 893 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:43:43.879536 893 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:43:43.913017 893 rpc_server.cc:307] RPC server started. Bound to: 127.0.194.193:37873
I20260926 03:43:43.913110 1020 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.194.193:37873 every 8 connection(s)
I20260926 03:43:43.914386 893 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790394221564982-779-0/minicluster-data/ts-0/data/info.pb
I20260926 03:43:43.919363 779 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu as pid 893
I20260926 03:43:43.919589 779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790394221564982-779-0/minicluster-data/ts-0/wal/instance
I20260926 03:43:43.920174 779 external_mini_cluster.cc:1428] Setting key d12fe2448216bbc057924188603b3513
Sep 26 03:43:43 dist-test-slave-n2th krb5kdc[786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394223, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.193@KRBTEST.COM for kudu/127.0.194.254@KRBTEST.COM
I20260926 03:43:43.937106 1023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:43.917020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.193:33963 (local address 127.0.194.254:43495)
0926 03:43:43.917471 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0926 03:43:43.917475 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:43:43.918457 (+ 982us) server_negotiation.cc:407] Connection header received
0926 03:43:43.920513 (+ 2056us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:43:43.920518 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:43:43.920612 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:43:43.920729 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:43:43.922873 (+ 2144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:43.923563 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:43.925126 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:43.925541 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:43.925690 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:43:43.929909 (+ 4219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:43:43.929919 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:43:43.929927 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:43:43.929974 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:43:43.933620 (+ 3646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:43:43.934476 (+ 856us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:43:43.934485 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:43:43.934490 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:43:43.934661 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:43:43.935149 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:43:43.935156 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:43:43.935162 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:43:43.935542 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:43:43.935711 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0926 03:43:43.936413 (+ 702us) server_negotiation.cc:299] Negotiation successful
0926 03:43:43.936668 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":219,"threads_started":1}
WARNING: no policy specified for kudu/127.0.194.194@KRBTEST.COM; defaulting to no policy
I20260926 03:43:43.937821 1024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:43.917605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:43495 (local address 127.0.194.193:33963)
0926 03:43:43.918289 (+ 684us) negotiation.cc:107] Waiting for socket to connect
0926 03:43:43.918331 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 03:43:43.920192 (+ 1861us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:43:43.921049 (+ 857us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:43:43.921065 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:43:43.921927 (+ 862us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:43:43.922656 (+ 729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:43:43.922675 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:43:43.923731 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:43:43.923738 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:43:43.924975 (+ 1237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:43:43.924987 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:43:43.925798 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:43:43.925805 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:43:43.926021 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:43:43.926721 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:43:43.926799 (+ 78us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:43:43.929585 (+ 2786us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:43:43.933831 (+ 4246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:43:43.933841 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:43:43.933856 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:43:43.934306 (+ 450us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:43:43.934826 (+ 520us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:43:43.934830 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:43:43.934832 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:43:43.935014 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:43:43.935670 (+ 656us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:43:43.935676 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:43:43.936008 (+ 332us) client_negotiation.cc:769] Sending connection context
0926 03:43:43.936230 (+ 222us) client_negotiation.cc:240] Negotiation successful
0926 03:43:43.936644 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":262,"threads_started":1}
I20260926 03:43:43.939601 1021 heartbeater.cc:347] Connected to a master server at 127.0.194.254:43495
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.194.194@KRBTEST.COM" created.
I20260926 03:43:43.939908 1021 heartbeater.cc:464] Registering TS with master...
I20260926 03:43:43.940670 1021 heartbeater.cc:511] Master 127.0.194.254:43495 requested a full tablet report, sending...
I20260926 03:43:43.942910 831 ts_manager.cc:194] Registered new tserver with Master: 5e1ed814ab1846f081466f92bb4b7ed7 (127.0.194.193:37873)
I20260926 03:43:43.944963 831 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.194.193@KRBTEST.COM'} at 127.0.194.193:33963
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.194.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790394221564982-779-0/krb5kdc/kudu_127.0.194.194.keytab.
Entry for principal kudu/127.0.194.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790394221564982-779-0/krb5kdc/kudu_127.0.194.194.keytab.
I20260926 03:43:43.959664 779 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
/tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790394221564982-779-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790394221564982-779-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790394221564982-779-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790394221564982-779-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.194.194:0
--local_ip_for_outbound_sockets=127.0.194.194
--webserver_interface=127.0.194.194
--webserver_port=0
--tserver_master_addrs=127.0.194.254:43495
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790394221564982-779-0/krb5kdc/kudu_127.0.194.194.keytab
--principal=kudu/127.0.194.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.194.212:39905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790394221564982-779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790394221564982-779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790394221564982-779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:43:44.281718 1027 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:43:44.282073 1027 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:43:44.282240 1027 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:43:44.289655 1027 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:43:44.289836 1027 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:43:44.290032 1027 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.194.194
I20260926 03:43:44.299901 1027 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.194.212:39905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790394221564982-779-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790394221564982-779-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790394221564982-779-0/krb5kdc/kudu_127.0.194.194.keytab
--principal=kudu/127.0.194.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.194.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790394221564982-779-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.194.194
--webserver_port=0
--tserver_master_addrs=127.0.194.254:43495
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.194.194
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790394221564982-779-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:43:44.301291 1027 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:43:44.302820 1027 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:43:44.315608 1033 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:43:44.315707 1032 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:43:44.316407 1035 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:43:44.317224 1027 server_base.cc:1034] running on GCE node
I20260926 03:43:44.317868 1027 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:43:44.318841 1027 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:43:44.320065 1027 hybrid_clock.cc:648] HybridClock initialized: now 1790394224320004 us; error 60 us; skew 500 ppm
I20260926 03:43:44.320390 1027 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:43:44 dist-test-slave-n2th krb5kdc[786](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394224, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:43:44.323590 1027 init.cc:377] Logged in from keytab as kudu/127.0.194.194@KRBTEST.COM (short username kudu)
I20260926 03:43:44.325153 1027 webserver.cc:460] Webserver started at http://127.0.194.194:43251/ using document root <none> and password file <none>
I20260926 03:43:44.325873 1027 fs_manager.cc:359] Metadata directory not provided
I20260926 03:43:44.326020 1027 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:43:44.326324 1027 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:43:44.329276 1027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790394221564982-779-0/minicluster-data/ts-1/data/instance:
uuid: "af8f833ecf1f40d7bb9a39fae9d6401b"
format_stamp: "Formatted at 2026-09-26 03:43:44 on dist-test-slave-n2th"
server_key: "8e0fce233173a6b48574c0ae552b848d"
server_key_iv: "b8194ddcef4bb9630c758becfdf00295"
server_key_version: "encryptionkey@0"
I20260926 03:43:44.330086 1027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790394221564982-779-0/minicluster-data/ts-1/wal/instance:
uuid: "af8f833ecf1f40d7bb9a39fae9d6401b"
format_stamp: "Formatted at 2026-09-26 03:43:44 on dist-test-slave-n2th"
server_key: "8e0fce233173a6b48574c0ae552b848d"
server_key_iv: "b8194ddcef4bb9630c758becfdf00295"
server_key_version: "encryptionkey@0"
I20260926 03:43:44.335680 1027 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260926 03:43:44.340088 1042 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:43:44.341811 1027 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260926 03:43:44.342026 1027 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790394221564982-779-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790394221564982-779-0/minicluster-data/ts-1/wal
uuid: "af8f833ecf1f40d7bb9a39fae9d6401b"
format_stamp: "Formatted at 2026-09-26 03:43:44 on dist-test-slave-n2th"
server_key: "8e0fce233173a6b48574c0ae552b848d"
server_key_iv: "b8194ddcef4bb9630c758becfdf00295"
server_key_version: "encryptionkey@0"
I20260926 03:43:44.342270 1027 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790394221564982-779-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790394221564982-779-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790394221564982-779-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:43:44.363246 1027 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:43:44.367744 1027 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:43:44.368045 1027 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:43:44.369025 1027 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:43:44.370354 1027 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:43:44.370434 1027 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:43:44.370618 1027 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:43:44.370668 1027 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:43:44.404388 1027 rpc_server.cc:307] RPC server started. Bound to: 127.0.194.194:42021
I20260926 03:43:44.404438 1154 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.194.194:42021 every 8 connection(s)
I20260926 03:43:44.405723 1027 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790394221564982-779-0/minicluster-data/ts-1/data/info.pb
I20260926 03:43:44.406311 779 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu as pid 1027
I20260926 03:43:44.406522 779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790394221564982-779-0/minicluster-data/ts-1/wal/instance
I20260926 03:43:44.407095 779 external_mini_cluster.cc:1428] Setting key a425e4091b598c9eaf5eea847f01aea7
Sep 26 03:43:44 dist-test-slave-n2th krb5kdc[786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394224, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.194@KRBTEST.COM for kudu/127.0.194.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.194.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.194.195@KRBTEST.COM" created.
I20260926 03:43:44.426929 1023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:44.408433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.194:49893 (local address 127.0.194.254:43495)
0926 03:43:44.408608 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0926 03:43:44.408613 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:43:44.409806 (+ 1193us) server_negotiation.cc:407] Connection header received
0926 03:43:44.411699 (+ 1893us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:43:44.411706 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:43:44.411793 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:43:44.411911 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:43:44.413465 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:44.414183 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:44.415804 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:44.416191 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:44.416320 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:43:44.420118 (+ 3798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:43:44.420128 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:43:44.420136 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:43:44.420185 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:43:44.423505 (+ 3320us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:43:44.424310 (+ 805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:43:44.424314 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:43:44.424317 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:43:44.424448 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:43:44.424851 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:43:44.424859 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:43:44.424863 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:43:44.425150 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:43:44.425332 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0926 03:43:44.426146 (+ 814us) server_negotiation.cc:299] Negotiation successful
0926 03:43:44.426407 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 03:43:44.427583 1157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:44.408954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:43495 (local address 127.0.194.194:49893)
0926 03:43:44.409650 (+ 696us) negotiation.cc:107] Waiting for socket to connect
0926 03:43:44.409693 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 03:43:44.411426 (+ 1733us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:43:44.412160 (+ 734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:43:44.412168 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:43:44.412694 (+ 526us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:43:44.413287 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:43:44.413303 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:43:44.414330 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:43:44.414343 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:43:44.415636 (+ 1293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:43:44.415647 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:43:44.416423 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:43:44.416429 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:43:44.416654 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:43:44.417340 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:43:44.417365 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:43:44.419843 (+ 2478us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:43:44.423671 (+ 3828us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:43:44.423678 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:43:44.423691 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:43:44.424192 (+ 501us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:43:44.424542 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:43:44.424545 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:43:44.424548 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:43:44.424719 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:43:44.425254 (+ 535us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:43:44.425260 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:43:44.425733 (+ 473us) client_negotiation.cc:769] Sending connection context
0926 03:43:44.426004 (+ 271us) client_negotiation.cc:240] Negotiation successful
0926 03:43:44.426398 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":289,"threads_started":1}
I20260926 03:43:44.429174 1155 heartbeater.cc:347] Connected to a master server at 127.0.194.254:43495
I20260926 03:43:44.429512 1155 heartbeater.cc:464] Registering TS with master...
I20260926 03:43:44.430297 1155 heartbeater.cc:511] Master 127.0.194.254:43495 requested a full tablet report, sending...
I20260926 03:43:44.431813 831 ts_manager.cc:194] Registered new tserver with Master: af8f833ecf1f40d7bb9a39fae9d6401b (127.0.194.194:42021)
I20260926 03:43:44.432622 831 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.194.194@KRBTEST.COM'} at 127.0.194.194:49893
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.194.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790394221564982-779-0/krb5kdc/kudu_127.0.194.195.keytab.
Entry for principal kudu/127.0.194.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790394221564982-779-0/krb5kdc/kudu_127.0.194.195.keytab.
I20260926 03:43:44.446244 779 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
/tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790394221564982-779-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790394221564982-779-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790394221564982-779-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790394221564982-779-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.194.195:0
--local_ip_for_outbound_sockets=127.0.194.195
--webserver_interface=127.0.194.195
--webserver_port=0
--tserver_master_addrs=127.0.194.254:43495
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790394221564982-779-0/krb5kdc/kudu_127.0.194.195.keytab
--principal=kudu/127.0.194.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.194.212:39905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790394221564982-779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790394221564982-779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790394221564982-779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:43:44.787431 1160 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:43:44.787758 1160 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:43:44.787921 1160 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:43:44.795190 1160 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:43:44.795356 1160 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:43:44.795534 1160 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.194.195
I20260926 03:43:44.805701 1160 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.194.212:39905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790394221564982-779-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790394221564982-779-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790394221564982-779-0/krb5kdc/kudu_127.0.194.195.keytab
--principal=kudu/127.0.194.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.194.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790394221564982-779-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.194.195
--webserver_port=0
--tserver_master_addrs=127.0.194.254:43495
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.194.195
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790394221564982-779-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:43:44.807127 1160 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:43:44.808485 1160 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:43:44.821560 1165 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:43:44.821560 1168 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:43:44.821594 1166 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:43:44.822382 1160 server_base.cc:1034] running on GCE node
I20260926 03:43:44.823181 1160 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:43:44.824177 1160 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:43:44.825399 1160 hybrid_clock.cc:648] HybridClock initialized: now 1790394224825354 us; error 62 us; skew 500 ppm
I20260926 03:43:44.825729 1160 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:43:44 dist-test-slave-n2th krb5kdc[786](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394224, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:43:44.828836 1160 init.cc:377] Logged in from keytab as kudu/127.0.194.195@KRBTEST.COM (short username kudu)
I20260926 03:43:44.830288 1160 webserver.cc:460] Webserver started at http://127.0.194.195:40869/ using document root <none> and password file <none>
I20260926 03:43:44.831070 1160 fs_manager.cc:359] Metadata directory not provided
I20260926 03:43:44.831218 1160 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:43:44.831540 1160 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:43:44.834435 1160 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790394221564982-779-0/minicluster-data/ts-2/data/instance:
uuid: "8a93b66990a34a9b84505db0f8b69964"
format_stamp: "Formatted at 2026-09-26 03:43:44 on dist-test-slave-n2th"
server_key: "dcc1de08967c7acd483d6654e8fdd5e2"
server_key_iv: "9f6443c911648e5afaa09c5e9a6d53f1"
server_key_version: "encryptionkey@0"
I20260926 03:43:44.835311 1160 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790394221564982-779-0/minicluster-data/ts-2/wal/instance:
uuid: "8a93b66990a34a9b84505db0f8b69964"
format_stamp: "Formatted at 2026-09-26 03:43:44 on dist-test-slave-n2th"
server_key: "dcc1de08967c7acd483d6654e8fdd5e2"
server_key_iv: "9f6443c911648e5afaa09c5e9a6d53f1"
server_key_version: "encryptionkey@0"
I20260926 03:43:44.840574 1160 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 03:43:44.844645 1175 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:43:44.846084 1160 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 03:43:44.846297 1160 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790394221564982-779-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790394221564982-779-0/minicluster-data/ts-2/wal
uuid: "8a93b66990a34a9b84505db0f8b69964"
format_stamp: "Formatted at 2026-09-26 03:43:44 on dist-test-slave-n2th"
server_key: "dcc1de08967c7acd483d6654e8fdd5e2"
server_key_iv: "9f6443c911648e5afaa09c5e9a6d53f1"
server_key_version: "encryptionkey@0"
I20260926 03:43:44.846529 1160 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790394221564982-779-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790394221564982-779-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790394221564982-779-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:43:44.884467 1160 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:43:44.888729 1160 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:43:44.889026 1160 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:43:44.889974 1160 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:43:44.891352 1160 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:43:44.891450 1160 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:43:44.891606 1160 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:43:44.891680 1160 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:43:44.925542 1160 rpc_server.cc:307] RPC server started. Bound to: 127.0.194.195:40335
I20260926 03:43:44.925632 1287 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.194.195:40335 every 8 connection(s)
I20260926 03:43:44.926898 1160 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790394221564982-779-0/minicluster-data/ts-2/data/info.pb
I20260926 03:43:44.934931 779 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu as pid 1160
I20260926 03:43:44.935129 779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790394221564982-779-0/minicluster-data/ts-2/wal/instance
I20260926 03:43:44.935564 779 external_mini_cluster.cc:1428] Setting key f6ebf422bc5650e762174c7ec2d7ffc8
Sep 26 03:43:44 dist-test-slave-n2th krb5kdc[786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394224, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.195@KRBTEST.COM for kudu/127.0.194.254@KRBTEST.COM
I20260926 03:43:44.948182 1021 heartbeater.cc:503] Master 127.0.194.254:43495 was elected leader, sending a full tablet report...
I20260926 03:43:44.948604 1290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:44.929647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.195:40259 (local address 127.0.194.254:43495)
0926 03:43:44.930118 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0926 03:43:44.930123 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:43:44.931021 (+ 898us) server_negotiation.cc:407] Connection header received
0926 03:43:44.932879 (+ 1858us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:43:44.932885 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:43:44.932975 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:43:44.933102 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:43:44.934619 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:44.936175 (+ 1556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:44.938469 (+ 2294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:44.939223 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:44.939362 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:43:44.942856 (+ 3494us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:43:44.942861 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:43:44.942865 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:43:44.942899 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:43:44.945279 (+ 2380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:43:44.945997 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:43:44.946008 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:43:44.946010 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:43:44.946118 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:43:44.946557 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:43:44.946560 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:43:44.946562 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:43:44.946889 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:43:44.946989 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0926 03:43:44.947964 (+ 975us) server_negotiation.cc:299] Negotiation successful
0926 03:43:44.948187 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":253,"threads_started":1}
I20260926 03:43:44.949587 1291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:44.930131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:43495 (local address 127.0.194.195:40259)
0926 03:43:44.930855 (+ 724us) negotiation.cc:107] Waiting for socket to connect
0926 03:43:44.930905 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 03:43:44.932576 (+ 1671us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:43:44.933339 (+ 763us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:43:44.933347 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:43:44.933854 (+ 507us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:43:44.934408 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:43:44.934424 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:43:44.936363 (+ 1939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:43:44.936369 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:43:44.938316 (+ 1947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:43:44.938329 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:43:44.939467 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:43:44.939476 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:43:44.939780 (+ 304us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:43:44.940430 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:43:44.940455 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:43:44.942588 (+ 2133us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:43:44.945435 (+ 2847us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:43:44.945443 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:43:44.945458 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:43:44.945895 (+ 437us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:43:44.946215 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:43:44.946218 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:43:44.946220 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:43:44.946456 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:43:44.947028 (+ 572us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:43:44.947035 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:43:44.947392 (+ 357us) client_negotiation.cc:769] Sending connection context
0926 03:43:44.947741 (+ 349us) client_negotiation.cc:240] Negotiation successful
0926 03:43:44.948243 (+ 502us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":257,"threads_started":1}
I20260926 03:43:44.951488 1288 heartbeater.cc:347] Connected to a master server at 127.0.194.254:43495
I20260926 03:43:44.951818 1288 heartbeater.cc:464] Registering TS with master...
I20260926 03:43:44.952621 1288 heartbeater.cc:511] Master 127.0.194.254:43495 requested a full tablet report, sending...
I20260926 03:43:44.954042 830 ts_manager.cc:194] Registered new tserver with Master: 8a93b66990a34a9b84505db0f8b69964 (127.0.194.195:40335)
I20260926 03:43:44.955051 830 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.194.195@KRBTEST.COM'} at 127.0.194.195:40259
I20260926 03:43:44.964710 779 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 03:43:44 dist-test-slave-n2th krb5kdc[786](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394224, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 03:43:45 dist-test-slave-n2th krb5kdc[786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394224, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.194.193@KRBTEST.COM
I20260926 03:43:45.010474 1296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:44.994825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44314 (local address 127.0.194.193:37873)
0926 03:43:44.995525 (+ 700us) server_negotiation.cc:206] Beginning negotiation
0926 03:43:44.995534 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:43:44.995565 (+ 31us) server_negotiation.cc:407] Connection header received
0926 03:43:44.995694 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:43:44.995707 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:43:44.995923 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:43:44.996086 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:43:44.997589 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:44.998595 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:44.999913 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:45.000485 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:45.000625 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:43:45.004718 (+ 4093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:43:45.004731 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:43:45.004746 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:43:45.004794 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:43:45.007550 (+ 2756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:43:45.008289 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:43:45.008297 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:43:45.008299 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:43:45.008423 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:43:45.008860 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:43:45.008864 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:43:45.008866 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:43:45.009210 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:43:45.009524 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0926 03:43:45.009828 (+ 304us) server_negotiation.cc:299] Negotiation successful
0926 03:43:45.010104 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":516,"thread_start_us":323,"threads_started":1}
W20260926 03:43:45.011848 936 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44314
Sep 26 03:43:45 dist-test-slave-n2th krb5kdc[786](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394225, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 03:43:45 dist-test-slave-n2th krb5kdc[786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394225, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.194.193@KRBTEST.COM
I20260926 03:43:45.055896 1296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:45.041672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44326 (local address 127.0.194.193:37873)
0926 03:43:45.041916 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0926 03:43:45.041922 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:43:45.042142 (+ 220us) server_negotiation.cc:407] Connection header received
0926 03:43:45.042301 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:43:45.042306 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:43:45.042370 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:43:45.042456 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:43:45.043888 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:45.044562 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:45.045772 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:45.046208 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:45.046319 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:43:45.050461 (+ 4142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:43:45.050466 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:43:45.050470 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:43:45.050507 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:43:45.053632 (+ 3125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:43:45.054267 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:43:45.054271 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:43:45.054274 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:43:45.054410 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:43:45.054718 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:43:45.054721 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:43:45.054723 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:43:45.055037 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:43:45.055211 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0926 03:43:45.055485 (+ 274us) server_negotiation.cc:299] Negotiation successful
0926 03:43:45.055650 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260926 03:43:45.057476 779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu pid 893
I20260926 03:43:45.070696 1296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:45.058397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44338 (local address 127.0.194.193:37873)
0926 03:43:45.058612 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0926 03:43:45.058622 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 03:43:45.058958 (+ 336us) server_negotiation.cc:407] Connection header received
0926 03:43:45.059079 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:43:45.059082 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:43:45.059136 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:43:45.059210 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:43:45.060338 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:45.061016 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:45.062290 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:45.062687 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:45.062863 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:43:45.065307 (+ 2444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:43:45.065311 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:43:45.065315 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:43:45.065345 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:43:45.068020 (+ 2675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:43:45.068703 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:43:45.068708 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:43:45.068710 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:43:45.068852 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:43:45.069224 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:43:45.069229 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:43:45.069231 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:43:45.069547 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:43:45.069823 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0926 03:43:45.070026 (+ 203us) server_negotiation.cc:299] Negotiation successful
0926 03:43:45.070189 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 03:43:45.071471 1016 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:43:45.153553 779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu with pid 893
I20260926 03:43:45.170574 779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu pid 1027
Sep 26 03:43:45 dist-test-slave-n2th krb5kdc[786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394225, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.194.194@KRBTEST.COM
I20260926 03:43:45.186225 1303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:45.171490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48342 (local address 127.0.194.194:42021)
0926 03:43:45.172023 (+ 533us) server_negotiation.cc:206] Beginning negotiation
0926 03:43:45.172030 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:43:45.172054 (+ 24us) server_negotiation.cc:407] Connection header received
0926 03:43:45.172113 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:43:45.172119 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:43:45.172322 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:43:45.172444 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:43:45.173552 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:45.174513 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:45.175731 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:45.176257 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:45.176349 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:43:45.180571 (+ 4222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:43:45.180582 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:43:45.180596 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:43:45.180628 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:43:45.183541 (+ 2913us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:43:45.184202 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:43:45.184208 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:43:45.184210 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:43:45.184329 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:43:45.184730 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:43:45.184734 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:43:45.184735 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:43:45.185132 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:43:45.185363 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0926 03:43:45.185634 (+ 271us) server_negotiation.cc:299] Negotiation successful
0926 03:43:45.185869 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":396,"thread_start_us":294,"threads_started":1}
I20260926 03:43:45.187501 1150 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:43:45.268944 779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu with pid 1027
I20260926 03:43:45.290213 779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu pid 1160
Sep 26 03:43:45 dist-test-slave-n2th krb5kdc[786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394225, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.194.195@KRBTEST.COM
I20260926 03:43:45.307096 1305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:45.291365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49676 (local address 127.0.194.195:40335)
0926 03:43:45.291954 (+ 589us) server_negotiation.cc:206] Beginning negotiation
0926 03:43:45.291961 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:43:45.291991 (+ 30us) server_negotiation.cc:407] Connection header received
0926 03:43:45.292063 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:43:45.292069 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:43:45.292296 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:43:45.292483 (+ 187us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:43:45.293733 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:45.294822 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:45.296103 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:45.296680 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:45.296821 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:43:45.301080 (+ 4259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:43:45.301094 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:43:45.301110 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:43:45.301153 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:43:45.304207 (+ 3054us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:43:45.304935 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:43:45.304943 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:43:45.304946 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:43:45.305071 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:43:45.305454 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:43:45.305459 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:43:45.305465 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:43:45.305810 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:43:45.306124 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0926 03:43:45.306442 (+ 318us) server_negotiation.cc:299] Negotiation successful
0926 03:43:45.306720 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":445,"thread_start_us":290,"threads_started":1}
I20260926 03:43:45.308379 1283 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:43:45.392925 779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu with pid 1160
I20260926 03:43:45.409211 779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu pid 799
I20260926 03:43:45.410413 861 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:43:45.512132 779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu with pid 799
2026-09-26T03:43:45Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (3911 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790394221564982-779-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:43:45 dist-test-slave-n2th krb5kdc[1310](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:43:45 dist-test-slave-n2th krb5kdc[1310](info): set up 2 sockets
Sep 26 03:43:45 dist-test-slave-n2th krb5kdc[1310](info): commencing operation
krb5kdc: starting...
W20260926 03:43:46.602665 779 mini_kdc.cc:121] Time spent starting KDC: real 1.049s user 0.000s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790394221564982-779-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790394221564982-779-0/krb5kdc/test-user.keytab.
Sep 26 03:43:46 dist-test-slave-n2th krb5kdc[1310](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394226, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:43:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:43:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.194.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.194.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.194.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790394221564982-779-0/krb5kdc/kudu_127.0.194.254.keytab.
Entry for principal kudu/127.0.194.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790394221564982-779-0/krb5kdc/kudu_127.0.194.254.keytab.
I20260926 03:43:46.789901 779 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
/tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790394221564982-779-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790394221564982-779-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790394221564982-779-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790394221564982-779-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.194.254:40529
--webserver_interface=127.0.194.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790394221564982-779-0/krb5kdc/kudu_127.0.194.254.keytab
--principal=kudu/127.0.194.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.194.212:40469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.194.254:40529
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790394221564982-779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790394221564982-779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790394221564982-779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:43:47.113718 1323 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:43:47.114099 1323 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:43:47.114287 1323 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:43:47.121693 1323 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:43:47.121883 1323 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:43:47.121973 1323 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:43:47.122080 1323 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:43:47.122170 1323 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:43:47.132517 1323 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.194.212:40469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790394221564982-779-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790394221564982-779-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.194.254:40529
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790394221564982-779-0/krb5kdc/kudu_127.0.194.254.keytab
--principal=kudu/127.0.194.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.194.254:40529
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790394221564982-779-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.194.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790394221564982-779-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:43:47.134009 1323 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:43:47.135418 1323 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:43:47.147487 1331 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:43:47.147589 1329 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:43:47.147549 1328 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:43:47.149142 1323 server_base.cc:1034] running on GCE node
I20260926 03:43:47.149969 1323 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:43:47.151582 1323 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:43:47.152837 1323 hybrid_clock.cc:648] HybridClock initialized: now 1790394227152784 us; error 46 us; skew 500 ppm
I20260926 03:43:47.153208 1323 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:43:47 dist-test-slave-n2th krb5kdc[1310](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394227, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:43:47.163580 1323 init.cc:377] Logged in from keytab as kudu/127.0.194.254@KRBTEST.COM (short username kudu)
I20260926 03:43:47.165273 1323 webserver.cc:460] Webserver started at http://127.0.194.254:42905/ using document root <none> and password file <none>
I20260926 03:43:47.166030 1323 fs_manager.cc:359] Metadata directory not provided
I20260926 03:43:47.166133 1323 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:43:47.166391 1323 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:43:47.169500 1323 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790394221564982-779-0/minicluster-data/master-0/data/instance:
uuid: "86f2590a6a794c48bc6077a21ef08ac7"
format_stamp: "Formatted at 2026-09-26 03:43:47 on dist-test-slave-n2th"
server_key: "a8d012a00a2adc4e0ebf0ade4d5d877a"
server_key_iv: "c6f160c19fa117b22334a6a4d86f7c21"
server_key_version: "encryptionkey@0"
I20260926 03:43:47.170228 1323 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790394221564982-779-0/minicluster-data/master-0/wal/instance:
uuid: "86f2590a6a794c48bc6077a21ef08ac7"
format_stamp: "Formatted at 2026-09-26 03:43:47 on dist-test-slave-n2th"
server_key: "a8d012a00a2adc4e0ebf0ade4d5d877a"
server_key_iv: "c6f160c19fa117b22334a6a4d86f7c21"
server_key_version: "encryptionkey@0"
I20260926 03:43:47.175696 1323 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260926 03:43:47.179275 1338 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:43:47.180727 1323 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260926 03:43:47.180905 1323 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790394221564982-779-0/minicluster-data/master-0/data,/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790394221564982-779-0/minicluster-data/master-0/wal
uuid: "86f2590a6a794c48bc6077a21ef08ac7"
format_stamp: "Formatted at 2026-09-26 03:43:47 on dist-test-slave-n2th"
server_key: "a8d012a00a2adc4e0ebf0ade4d5d877a"
server_key_iv: "c6f160c19fa117b22334a6a4d86f7c21"
server_key_version: "encryptionkey@0"
I20260926 03:43:47.181123 1323 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790394221564982-779-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790394221564982-779-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790394221564982-779-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:43:47.196332 1323 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:43:47.200484 1323 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:43:47.200794 1323 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:43:47.219570 1323 rpc_server.cc:307] RPC server started. Bound to: 127.0.194.254:40529
I20260926 03:43:47.219594 1389 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.194.254:40529 every 8 connection(s)
I20260926 03:43:47.221050 1323 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790394221564982-779-0/minicluster-data/master-0/data/info.pb
I20260926 03:43:47.224829 1390 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:43:47.227289 779 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu as pid 1323
I20260926 03:43:47.227528 779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790394221564982-779-0/minicluster-data/master-0/wal/instance
I20260926 03:43:47.228332 779 external_mini_cluster.cc:1428] Setting key 82fa388a2000f664249520f46777ad50
I20260926 03:43:47.237069 1390 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86f2590a6a794c48bc6077a21ef08ac7: Bootstrap starting.
I20260926 03:43:47.240360 1390 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 86f2590a6a794c48bc6077a21ef08ac7: Neither blocks nor log segments found. Creating new log.
I20260926 03:43:47.241389 1390 log.cc:824] T 00000000000000000000000000000000 P 86f2590a6a794c48bc6077a21ef08ac7: Log is configured to *not* fsync() on all Append() calls
Sep 26 03:43:47 dist-test-slave-n2th krb5kdc[1310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394226, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.194.254@KRBTEST.COM
I20260926 03:43:47.244894 1390 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86f2590a6a794c48bc6077a21ef08ac7: No bootstrap required, opened a new log
I20260926 03:43:47.251533 1393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:47.230543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41314 (local address 127.0.194.254:40529)
0926 03:43:47.231407 (+ 864us) server_negotiation.cc:206] Beginning negotiation
0926 03:43:47.231420 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 03:43:47.231459 (+ 39us) server_negotiation.cc:407] Connection header received
0926 03:43:47.233148 (+ 1689us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:43:47.233169 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:43:47.233574 (+ 405us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:43:47.234009 (+ 435us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:43:47.235605 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:47.236919 (+ 1314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:47.238248 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:47.239068 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:47.239269 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:43:47.243740 (+ 4471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:43:47.243750 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:43:47.243766 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:43:47.243808 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:43:47.247069 (+ 3261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:43:47.247959 (+ 890us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:43:47.247966 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:43:47.247976 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:43:47.248144 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:43:47.248547 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:43:47.248550 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:43:47.248552 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:43:47.249117 (+ 565us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:43:47.249318 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0926 03:43:47.249917 (+ 599us) server_negotiation.cc:299] Negotiation successful
0926 03:43:47.250345 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":648,"thread_start_us":393,"threads_started":1}
I20260926 03:43:47.253984 1390 raft_consensus.cc:348] T 00000000000000000000000000000000 P 86f2590a6a794c48bc6077a21ef08ac7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86f2590a6a794c48bc6077a21ef08ac7" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 40529 } }
I20260926 03:43:47.254391 1390 raft_consensus.cc:374] T 00000000000000000000000000000000 P 86f2590a6a794c48bc6077a21ef08ac7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:43:47.254523 1390 raft_consensus.cc:729] T 00000000000000000000000000000000 P 86f2590a6a794c48bc6077a21ef08ac7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86f2590a6a794c48bc6077a21ef08ac7, State: Initialized, Role: FOLLOWER
I20260926 03:43:47.255127 1390 consensus_queue.cc:260] T 00000000000000000000000000000000 P 86f2590a6a794c48bc6077a21ef08ac7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86f2590a6a794c48bc6077a21ef08ac7" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 40529 } }
I20260926 03:43:47.255381 1390 raft_consensus.cc:388] T 00000000000000000000000000000000 P 86f2590a6a794c48bc6077a21ef08ac7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:43:47.255525 1390 raft_consensus.cc:482] T 00000000000000000000000000000000 P 86f2590a6a794c48bc6077a21ef08ac7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:43:47.255697 1390 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 86f2590a6a794c48bc6077a21ef08ac7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:43:47.257900 1390 raft_consensus.cc:504] T 00000000000000000000000000000000 P 86f2590a6a794c48bc6077a21ef08ac7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86f2590a6a794c48bc6077a21ef08ac7" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 40529 } }
I20260926 03:43:47.258455 1390 leader_election.cc:302] T 00000000000000000000000000000000 P 86f2590a6a794c48bc6077a21ef08ac7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 86f2590a6a794c48bc6077a21ef08ac7; no voters:
I20260926 03:43:47.259146 1390 leader_election.cc:288] T 00000000000000000000000000000000 P 86f2590a6a794c48bc6077a21ef08ac7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:43:47.259233 1395 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 86f2590a6a794c48bc6077a21ef08ac7 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:43:47.260003 1395 raft_consensus.cc:686] T 00000000000000000000000000000000 P 86f2590a6a794c48bc6077a21ef08ac7 [term 1 LEADER]: Becoming Leader. State: Replica: 86f2590a6a794c48bc6077a21ef08ac7, State: Running, Role: LEADER
I20260926 03:43:47.260535 1395 consensus_queue.cc:237] T 00000000000000000000000000000000 P 86f2590a6a794c48bc6077a21ef08ac7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86f2590a6a794c48bc6077a21ef08ac7" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 40529 } }
I20260926 03:43:47.261361 1390 sys_catalog.cc:564] T 00000000000000000000000000000000 P 86f2590a6a794c48bc6077a21ef08ac7 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:43:47.263520 1396 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86f2590a6a794c48bc6077a21ef08ac7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "86f2590a6a794c48bc6077a21ef08ac7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86f2590a6a794c48bc6077a21ef08ac7" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 40529 } } }
I20260926 03:43:47.263614 1397 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86f2590a6a794c48bc6077a21ef08ac7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 86f2590a6a794c48bc6077a21ef08ac7. Latest consensus state: current_term: 1 leader_uuid: "86f2590a6a794c48bc6077a21ef08ac7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86f2590a6a794c48bc6077a21ef08ac7" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 40529 } } }
I20260926 03:43:47.264025 1396 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86f2590a6a794c48bc6077a21ef08ac7 [sys.catalog]: This master's current role is: LEADER
I20260926 03:43:47.264065 1397 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86f2590a6a794c48bc6077a21ef08ac7 [sys.catalog]: This master's current role is: LEADER
I20260926 03:43:47.265754 1403 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:43:47.271610 1403 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:43:47.281073 1403 catalog_manager.cc:1347] Generated new cluster ID: 14a0454874f24aa99a15b393654cc4c0
I20260926 03:43:47.281217 1403 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:43:47.295307 1403 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:43:47.296943 1403 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:43:47.305636 1403 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 86f2590a6a794c48bc6077a21ef08ac7: Generated new TSK 0
I20260926 03:43:47.306612 1403 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.194.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.194.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.194.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790394221564982-779-0/krb5kdc/kudu_127.0.194.193.keytab.
Entry for principal kudu/127.0.194.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790394221564982-779-0/krb5kdc/kudu_127.0.194.193.keytab.
I20260926 03:43:47.353907 779 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
/tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790394221564982-779-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790394221564982-779-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790394221564982-779-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790394221564982-779-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.194.193:0
--local_ip_for_outbound_sockets=127.0.194.193
--webserver_interface=127.0.194.193
--webserver_port=0
--tserver_master_addrs=127.0.194.254:40529
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790394221564982-779-0/krb5kdc/kudu_127.0.194.193.keytab
--principal=kudu/127.0.194.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.194.212:40469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790394221564982-779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790394221564982-779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790394221564982-779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:43:47.688997 1416 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:43:47.689347 1416 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:43:47.689532 1416 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:43:47.697218 1416 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:43:47.697360 1416 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:43:47.697551 1416 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.194.193
I20260926 03:43:47.708127 1416 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.194.212:40469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790394221564982-779-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790394221564982-779-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790394221564982-779-0/krb5kdc/kudu_127.0.194.193.keytab
--principal=kudu/127.0.194.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.194.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790394221564982-779-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.194.193
--webserver_port=0
--tserver_master_addrs=127.0.194.254:40529
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.194.193
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790394221564982-779-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:43:47.709546 1416 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:43:47.711079 1416 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:43:47.728461 1421 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:43:47.728507 1424 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:43:47.728531 1422 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:43:47.729460 1416 server_base.cc:1034] running on GCE node
I20260926 03:43:47.730222 1416 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:43:47.731385 1416 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:43:47.732594 1416 hybrid_clock.cc:648] HybridClock initialized: now 1790394227732523 us; error 62 us; skew 500 ppm
I20260926 03:43:47.732954 1416 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:43:47 dist-test-slave-n2th krb5kdc[1310](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394227, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:43:47.736299 1416 init.cc:377] Logged in from keytab as kudu/127.0.194.193@KRBTEST.COM (short username kudu)
I20260926 03:43:47.737864 1416 webserver.cc:460] Webserver started at http://127.0.194.193:35275/ using document root <none> and password file <none>
I20260926 03:43:47.738621 1416 fs_manager.cc:359] Metadata directory not provided
I20260926 03:43:47.738812 1416 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:43:47.739135 1416 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:43:47.742187 1416 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790394221564982-779-0/minicluster-data/ts-0/data/instance:
uuid: "31de5182459c47ada0a3ee8972924d61"
format_stamp: "Formatted at 2026-09-26 03:43:47 on dist-test-slave-n2th"
server_key: "371a2a78805d210a3ef4aad01581ff98"
server_key_iv: "e309091ab2ceb64ee8b2bab283376beb"
server_key_version: "encryptionkey@0"
I20260926 03:43:47.743062 1416 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790394221564982-779-0/minicluster-data/ts-0/wal/instance:
uuid: "31de5182459c47ada0a3ee8972924d61"
format_stamp: "Formatted at 2026-09-26 03:43:47 on dist-test-slave-n2th"
server_key: "371a2a78805d210a3ef4aad01581ff98"
server_key_iv: "e309091ab2ceb64ee8b2bab283376beb"
server_key_version: "encryptionkey@0"
I20260926 03:43:47.748592 1416 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 03:43:47.752458 1431 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:43:47.754062 1416 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 03:43:47.754252 1416 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790394221564982-779-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790394221564982-779-0/minicluster-data/ts-0/wal
uuid: "31de5182459c47ada0a3ee8972924d61"
format_stamp: "Formatted at 2026-09-26 03:43:47 on dist-test-slave-n2th"
server_key: "371a2a78805d210a3ef4aad01581ff98"
server_key_iv: "e309091ab2ceb64ee8b2bab283376beb"
server_key_version: "encryptionkey@0"
I20260926 03:43:47.754499 1416 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790394221564982-779-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790394221564982-779-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790394221564982-779-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:43:47.770615 1416 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:43:47.775085 1416 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:43:47.775391 1416 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:43:47.776368 1416 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:43:47.777737 1416 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:43:47.777851 1416 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:43:47.777997 1416 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:43:47.778071 1416 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:43:47.811755 1416 rpc_server.cc:307] RPC server started. Bound to: 127.0.194.193:45885
I20260926 03:43:47.811800 1543 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.194.193:45885 every 8 connection(s)
I20260926 03:43:47.813131 1416 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790394221564982-779-0/minicluster-data/ts-0/data/info.pb
I20260926 03:43:47.822113 779 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu as pid 1416
I20260926 03:43:47.822329 779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790394221564982-779-0/minicluster-data/ts-0/wal/instance
I20260926 03:43:47.823007 779 external_mini_cluster.cc:1428] Setting key 1d300052aa770b2014de80fa3fabd5b2
Sep 26 03:43:47 dist-test-slave-n2th krb5kdc[1310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394227, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.193@KRBTEST.COM for kudu/127.0.194.254@KRBTEST.COM
I20260926 03:43:47.833673 1546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:47.815769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.193:45579 (local address 127.0.194.254:40529)
0926 03:43:47.816185 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0926 03:43:47.816190 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:43:47.817218 (+ 1028us) server_negotiation.cc:407] Connection header received
0926 03:43:47.819129 (+ 1911us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:43:47.819134 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:43:47.819240 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:43:47.819365 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:43:47.820953 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:47.821736 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:47.823621 (+ 1885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:47.824051 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:47.824211 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:43:47.827758 (+ 3547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:43:47.827764 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:43:47.827767 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:43:47.827801 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:43:47.830491 (+ 2690us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:43:47.831301 (+ 810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:43:47.831307 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:43:47.831317 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:43:47.831432 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:43:47.831876 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:43:47.831881 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:43:47.831884 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:43:47.832152 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:43:47.832313 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0926 03:43:47.833035 (+ 722us) server_negotiation.cc:299] Negotiation successful
0926 03:43:47.833300 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":211,"threads_started":1}
I20260926 03:43:47.834707 1547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:47.816359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:40529 (local address 127.0.194.193:45579)
0926 03:43:47.817052 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0926 03:43:47.817095 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 03:43:47.818844 (+ 1749us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:43:47.819619 (+ 775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:43:47.819628 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:43:47.820173 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:43:47.820755 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:43:47.820782 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:43:47.821938 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:43:47.821942 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:43:47.823447 (+ 1505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:43:47.823459 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:43:47.824291 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:43:47.824299 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:43:47.824571 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:43:47.825273 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:43:47.825296 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:43:47.827508 (+ 2212us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:43:47.830642 (+ 3134us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:43:47.830652 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:43:47.830671 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:43:47.831169 (+ 498us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:43:47.831540 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:43:47.831546 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:43:47.831549 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:43:47.831754 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:43:47.832283 (+ 529us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:43:47.832292 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:43:47.832652 (+ 360us) client_negotiation.cc:769] Sending connection context
0926 03:43:47.832935 (+ 283us) client_negotiation.cc:240] Negotiation successful
0926 03:43:47.833497 (+ 562us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":247,"threads_started":1}
I20260926 03:43:47.836555 1544 heartbeater.cc:347] Connected to a master server at 127.0.194.254:40529
I20260926 03:43:47.836880 1544 heartbeater.cc:464] Registering TS with master...
I20260926 03:43:47.837633 1544 heartbeater.cc:511] Master 127.0.194.254:40529 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.194.194@KRBTEST.COM; defaulting to no policy
I20260926 03:43:47.839848 1355 ts_manager.cc:194] Registered new tserver with Master: 31de5182459c47ada0a3ee8972924d61 (127.0.194.193:45885)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.194.194@KRBTEST.COM" created.
I20260926 03:43:47.841358 1355 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.194.193@KRBTEST.COM'} at 127.0.194.193:45579
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.194.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790394221564982-779-0/krb5kdc/kudu_127.0.194.194.keytab.
Entry for principal kudu/127.0.194.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790394221564982-779-0/krb5kdc/kudu_127.0.194.194.keytab.
I20260926 03:43:47.866509 779 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
/tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790394221564982-779-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790394221564982-779-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790394221564982-779-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790394221564982-779-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.194.194:0
--local_ip_for_outbound_sockets=127.0.194.194
--webserver_interface=127.0.194.194
--webserver_port=0
--tserver_master_addrs=127.0.194.254:40529
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790394221564982-779-0/krb5kdc/kudu_127.0.194.194.keytab
--principal=kudu/127.0.194.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.194.212:40469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790394221564982-779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790394221564982-779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790394221564982-779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:43:48.202065 1550 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:43:48.202370 1550 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:43:48.202468 1550 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:43:48.210083 1550 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:43:48.210213 1550 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:43:48.210330 1550 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.194.194
I20260926 03:43:48.221552 1550 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.194.212:40469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790394221564982-779-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790394221564982-779-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790394221564982-779-0/krb5kdc/kudu_127.0.194.194.keytab
--principal=kudu/127.0.194.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.194.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790394221564982-779-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.194.194
--webserver_port=0
--tserver_master_addrs=127.0.194.254:40529
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.194.194
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790394221564982-779-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:43:48.223204 1550 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:43:48.224922 1550 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:43:48.240383 1555 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:43:48.240530 1556 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:43:48.241019 1558 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:43:48.241468 1550 server_base.cc:1034] running on GCE node
I20260926 03:43:48.242120 1550 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:43:48.243160 1550 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:43:48.244376 1550 hybrid_clock.cc:648] HybridClock initialized: now 1790394228244289 us; error 78 us; skew 500 ppm
I20260926 03:43:48.244725 1550 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:43:48 dist-test-slave-n2th krb5kdc[1310](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394228, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:43:48.247994 1550 init.cc:377] Logged in from keytab as kudu/127.0.194.194@KRBTEST.COM (short username kudu)
I20260926 03:43:48.249691 1550 webserver.cc:460] Webserver started at http://127.0.194.194:40193/ using document root <none> and password file <none>
I20260926 03:43:48.250599 1550 fs_manager.cc:359] Metadata directory not provided
I20260926 03:43:48.250706 1550 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:43:48.251060 1550 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:43:48.254101 1550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790394221564982-779-0/minicluster-data/ts-1/data/instance:
uuid: "330f33cc31b74e9baedebff5da38e79f"
format_stamp: "Formatted at 2026-09-26 03:43:48 on dist-test-slave-n2th"
server_key: "eacfa73b206595fa46df2bf4d32dcb70"
server_key_iv: "e495dd959ab4bf0aa341b966416a9817"
server_key_version: "encryptionkey@0"
I20260926 03:43:48.254941 1550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790394221564982-779-0/minicluster-data/ts-1/wal/instance:
uuid: "330f33cc31b74e9baedebff5da38e79f"
format_stamp: "Formatted at 2026-09-26 03:43:48 on dist-test-slave-n2th"
server_key: "eacfa73b206595fa46df2bf4d32dcb70"
server_key_iv: "e495dd959ab4bf0aa341b966416a9817"
server_key_version: "encryptionkey@0"
I20260926 03:43:48.260459 1550 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 03:43:48.264281 1565 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:43:48.265729 1550 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 03:43:48.265967 1550 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790394221564982-779-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790394221564982-779-0/minicluster-data/ts-1/wal
uuid: "330f33cc31b74e9baedebff5da38e79f"
format_stamp: "Formatted at 2026-09-26 03:43:48 on dist-test-slave-n2th"
server_key: "eacfa73b206595fa46df2bf4d32dcb70"
server_key_iv: "e495dd959ab4bf0aa341b966416a9817"
server_key_version: "encryptionkey@0"
I20260926 03:43:48.266176 1550 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790394221564982-779-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790394221564982-779-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790394221564982-779-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:43:48.277470 1550 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:43:48.281427 1550 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:43:48.281718 1550 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:43:48.282640 1550 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:43:48.283963 1550 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:43:48.284036 1550 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:43:48.284198 1550 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:43:48.284263 1550 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:43:48.317157 1550 rpc_server.cc:307] RPC server started. Bound to: 127.0.194.194:44819
I20260926 03:43:48.317204 1677 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.194.194:44819 every 8 connection(s)
I20260926 03:43:48.318490 1550 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790394221564982-779-0/minicluster-data/ts-1/data/info.pb
I20260926 03:43:48.325209 779 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu as pid 1550
I20260926 03:43:48.325428 779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790394221564982-779-0/minicluster-data/ts-1/wal/instance
I20260926 03:43:48.325863 779 external_mini_cluster.cc:1428] Setting key c0e58d110a4fbfd06cf501def907e15a
Sep 26 03:43:48 dist-test-slave-n2th krb5kdc[1310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394228, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.194@KRBTEST.COM for kudu/127.0.194.254@KRBTEST.COM
I20260926 03:43:48.339316 1546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:48.321127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.194:52111 (local address 127.0.194.254:40529)
0926 03:43:48.321317 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0926 03:43:48.321321 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:43:48.322522 (+ 1201us) server_negotiation.cc:407] Connection header received
0926 03:43:48.324403 (+ 1881us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:43:48.324408 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:43:48.324501 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:43:48.324613 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:43:48.326379 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:48.327187 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:48.328785 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:48.329213 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:48.329354 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:43:48.333333 (+ 3979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:43:48.333338 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:43:48.333341 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:43:48.333375 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:43:48.336033 (+ 2658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:43:48.336857 (+ 824us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:43:48.336862 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:43:48.336864 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:43:48.336990 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:43:48.337446 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:43:48.337453 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:43:48.337457 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:43:48.337728 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:43:48.337899 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0926 03:43:48.338698 (+ 799us) server_negotiation.cc:299] Negotiation successful
0926 03:43:48.338973 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260926 03:43:48.340350 1680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:48.321618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:40529 (local address 127.0.194.194:52111)
0926 03:43:48.322323 (+ 705us) negotiation.cc:107] Waiting for socket to connect
0926 03:43:48.322364 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 03:43:48.324109 (+ 1745us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:43:48.324871 (+ 762us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:43:48.324880 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:43:48.325510 (+ 630us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:43:48.326182 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:43:48.326202 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:43:48.327362 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:43:48.327366 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:43:48.328626 (+ 1260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:43:48.328642 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:43:48.329451 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:43:48.329458 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:43:48.329696 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:43:48.330494 (+ 798us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:43:48.330530 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:43:48.333068 (+ 2538us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:43:48.336199 (+ 3131us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:43:48.336210 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:43:48.336229 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:43:48.336690 (+ 461us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:43:48.337110 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:43:48.337117 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:43:48.337122 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:43:48.337325 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:43:48.337882 (+ 557us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:43:48.337890 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:43:48.338326 (+ 436us) client_negotiation.cc:769] Sending connection context
0926 03:43:48.338609 (+ 283us) client_negotiation.cc:240] Negotiation successful
0926 03:43:48.339207 (+ 598us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":258,"threads_started":1}
I20260926 03:43:48.342114 1678 heartbeater.cc:347] Connected to a master server at 127.0.194.254:40529
WARNING: no policy specified for kudu/127.0.194.195@KRBTEST.COM; defaulting to no policy
I20260926 03:43:48.342479 1678 heartbeater.cc:464] Registering TS with master...
I20260926 03:43:48.343297 1678 heartbeater.cc:511] Master 127.0.194.254:40529 requested a full tablet report, sending...
I20260926 03:43:48.344972 1355 ts_manager.cc:194] Registered new tserver with Master: 330f33cc31b74e9baedebff5da38e79f (127.0.194.194:44819)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.194.195@KRBTEST.COM" created.
I20260926 03:43:48.346117 1355 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.194.194@KRBTEST.COM'} at 127.0.194.194:52111
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.194.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790394221564982-779-0/krb5kdc/kudu_127.0.194.195.keytab.
Entry for principal kudu/127.0.194.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790394221564982-779-0/krb5kdc/kudu_127.0.194.195.keytab.
I20260926 03:43:48.367623 779 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
/tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790394221564982-779-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790394221564982-779-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790394221564982-779-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790394221564982-779-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.194.195:0
--local_ip_for_outbound_sockets=127.0.194.195
--webserver_interface=127.0.194.195
--webserver_port=0
--tserver_master_addrs=127.0.194.254:40529
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790394221564982-779-0/krb5kdc/kudu_127.0.194.195.keytab
--principal=kudu/127.0.194.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.194.212:40469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790394221564982-779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790394221564982-779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790394221564982-779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:43:48.692476 1683 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:43:48.692828 1683 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:43:48.692992 1683 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:43:48.700325 1683 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:43:48.700449 1683 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:43:48.700564 1683 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.194.195
I20260926 03:43:48.710321 1683 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.194.212:40469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790394221564982-779-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790394221564982-779-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790394221564982-779-0/krb5kdc/kudu_127.0.194.195.keytab
--principal=kudu/127.0.194.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.194.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790394221564982-779-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.194.195
--webserver_port=0
--tserver_master_addrs=127.0.194.254:40529
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.194.195
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790394221564982-779-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:43:48.711712 1683 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:43:48.713114 1683 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:43:48.726475 1689 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:43:48.726555 1688 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:43:48.726575 1691 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:43:48.727365 1683 server_base.cc:1034] running on GCE node
I20260926 03:43:48.727960 1683 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:43:48.728928 1683 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:43:48.730157 1683 hybrid_clock.cc:648] HybridClock initialized: now 1790394228730101 us; error 53 us; skew 500 ppm
I20260926 03:43:48.730487 1683 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:43:48 dist-test-slave-n2th krb5kdc[1310](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394228, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:43:48.733867 1683 init.cc:377] Logged in from keytab as kudu/127.0.194.195@KRBTEST.COM (short username kudu)
I20260926 03:43:48.735419 1683 webserver.cc:460] Webserver started at http://127.0.194.195:33739/ using document root <none> and password file <none>
I20260926 03:43:48.736142 1683 fs_manager.cc:359] Metadata directory not provided
I20260926 03:43:48.736292 1683 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:43:48.736637 1683 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:43:48.739693 1683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790394221564982-779-0/minicluster-data/ts-2/data/instance:
uuid: "47b865711c9b4d0485bf6dd46131bd40"
format_stamp: "Formatted at 2026-09-26 03:43:48 on dist-test-slave-n2th"
server_key: "aba8a231867f4a408575f883ddae3e0a"
server_key_iv: "73eb34dfae45bac9be3b503c972a6255"
server_key_version: "encryptionkey@0"
I20260926 03:43:48.740492 1683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790394221564982-779-0/minicluster-data/ts-2/wal/instance:
uuid: "47b865711c9b4d0485bf6dd46131bd40"
format_stamp: "Formatted at 2026-09-26 03:43:48 on dist-test-slave-n2th"
server_key: "aba8a231867f4a408575f883ddae3e0a"
server_key_iv: "73eb34dfae45bac9be3b503c972a6255"
server_key_version: "encryptionkey@0"
I20260926 03:43:48.745940 1683 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260926 03:43:48.749706 1698 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:43:48.751235 1683 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260926 03:43:48.751425 1683 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790394221564982-779-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790394221564982-779-0/minicluster-data/ts-2/wal
uuid: "47b865711c9b4d0485bf6dd46131bd40"
format_stamp: "Formatted at 2026-09-26 03:43:48 on dist-test-slave-n2th"
server_key: "aba8a231867f4a408575f883ddae3e0a"
server_key_iv: "73eb34dfae45bac9be3b503c972a6255"
server_key_version: "encryptionkey@0"
I20260926 03:43:48.751639 1683 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790394221564982-779-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790394221564982-779-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790394221564982-779-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:43:48.790431 1683 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:43:48.794764 1683 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:43:48.795073 1683 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:43:48.796025 1683 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:43:48.797382 1683 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:43:48.797497 1683 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:43:48.797645 1683 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:43:48.797719 1683 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:43:48.831568 1683 rpc_server.cc:307] RPC server started. Bound to: 127.0.194.195:39539
I20260926 03:43:48.831670 1810 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.194.195:39539 every 8 connection(s)
I20260926 03:43:48.832908 1683 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790394221564982-779-0/minicluster-data/ts-2/data/info.pb
I20260926 03:43:48.835687 779 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu as pid 1683
I20260926 03:43:48.835906 779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790394221564982-779-0/minicluster-data/ts-2/wal/instance
I20260926 03:43:48.836355 779 external_mini_cluster.cc:1428] Setting key 8182881bac55606aaf5fd2a9f7841420
I20260926 03:43:48.844429 1544 heartbeater.cc:503] Master 127.0.194.254:40529 was elected leader, sending a full tablet report...
Sep 26 03:43:48 dist-test-slave-n2th krb5kdc[1310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394228, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.195@KRBTEST.COM for kudu/127.0.194.254@KRBTEST.COM
I20260926 03:43:48.854331 1546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:48.835668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.195:51363 (local address 127.0.194.254:40529)
0926 03:43:48.835908 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0926 03:43:48.835916 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:43:48.837250 (+ 1334us) server_negotiation.cc:407] Connection header received
0926 03:43:48.839616 (+ 2366us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:43:48.839621 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:43:48.839712 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:43:48.839821 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:43:48.841519 (+ 1698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:48.842328 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:48.843998 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:48.844436 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:48.844571 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:43:48.848382 (+ 3811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:43:48.848387 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:43:48.848390 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:43:48.848421 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:43:48.851210 (+ 2789us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:43:48.851989 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:43:48.852000 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:43:48.852002 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:43:48.852134 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:43:48.852587 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:43:48.852596 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:43:48.852599 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:43:48.852862 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:43:48.853015 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0926 03:43:48.853760 (+ 745us) server_negotiation.cc:299] Negotiation successful
0926 03:43:48.853977 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260926 03:43:48.855501 1813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:48.836212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:40529 (local address 127.0.194.195:51363)
0926 03:43:48.837059 (+ 847us) negotiation.cc:107] Waiting for socket to connect
0926 03:43:48.837100 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 03:43:48.839289 (+ 2189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:43:48.840075 (+ 786us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:43:48.840083 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:43:48.840672 (+ 589us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:43:48.841254 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:43:48.841273 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:43:48.842486 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:43:48.842492 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:43:48.843842 (+ 1350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:43:48.843855 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:43:48.844676 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:43:48.844685 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:43:48.844964 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:43:48.845781 (+ 817us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:43:48.845816 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:43:48.848164 (+ 2348us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:43:48.851365 (+ 3201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:43:48.851375 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:43:48.851395 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:43:48.851857 (+ 462us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:43:48.852256 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:43:48.852261 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:43:48.852265 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:43:48.852463 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:43:48.852992 (+ 529us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:43:48.853000 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:43:48.853378 (+ 378us) client_negotiation.cc:769] Sending connection context
0926 03:43:48.853658 (+ 280us) client_negotiation.cc:240] Negotiation successful
0926 03:43:48.854221 (+ 563us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":585,"thread_start_us":371,"threads_started":1}
I20260926 03:43:48.856937 1811 heartbeater.cc:347] Connected to a master server at 127.0.194.254:40529
I20260926 03:43:48.857261 1811 heartbeater.cc:464] Registering TS with master...
I20260926 03:43:48.857992 1811 heartbeater.cc:511] Master 127.0.194.254:40529 requested a full tablet report, sending...
I20260926 03:43:48.859498 1355 ts_manager.cc:194] Registered new tserver with Master: 47b865711c9b4d0485bf6dd46131bd40 (127.0.194.195:39539)
I20260926 03:43:48.860343 1355 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.194.195@KRBTEST.COM'} at 127.0.194.195:51363
I20260926 03:43:48.864401 779 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 03:43:48.883174 1546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:48.870662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57778 (local address 127.0.194.254:40529)
0926 03:43:48.870919 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0926 03:43:48.870924 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:43:48.871487 (+ 563us) server_negotiation.cc:407] Connection header received
0926 03:43:48.871643 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:43:48.871647 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:43:48.871707 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:43:48.871807 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:43:48.873255 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:48.873969 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:48.875361 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:48.875784 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:48.875881 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:43:48.878229 (+ 2348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:43:48.878234 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:43:48.878238 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:43:48.878265 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:43:48.880703 (+ 2438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:43:48.881333 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:43:48.881336 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:43:48.881339 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:43:48.881457 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:43:48.881775 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:43:48.881778 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:43:48.881780 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:43:48.882144 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:43:48.882327 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0926 03:43:48.882606 (+ 279us) server_negotiation.cc:299] Negotiation successful
0926 03:43:48.882831 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260926 03:43:48.889622 1355 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57778:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 03:43:48.896445 1355 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 03:43:48.932915 1824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:48.908487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32774 (local address 127.0.194.193:45885)
0926 03:43:48.923563 (+ 15076us) server_negotiation.cc:206] Beginning negotiation
0926 03:43:48.923574 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 03:43:48.923605 (+ 31us) server_negotiation.cc:407] Connection header received
0926 03:43:48.923750 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:43:48.923762 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:43:48.924042 (+ 280us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:43:48.924284 (+ 242us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:43:48.925412 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:48.927036 (+ 1624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:48.929849 (+ 2813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:48.931403 (+ 1554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:48.931629 (+ 226us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:43:48.931879 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0926 03:43:48.932407 (+ 528us) server_negotiation.cc:299] Negotiation successful
0926 03:43:48.932579 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14887,"thread_start_us":14761,"threads_started":1}
I20260926 03:43:48.933107 1823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:48.908354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.193:45885 (local address 127.0.0.1:32774)
0926 03:43:48.909019 (+ 665us) negotiation.cc:107] Waiting for socket to connect
0926 03:43:48.909053 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0926 03:43:48.909217 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:43:48.924288 (+ 15071us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:43:48.924294 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:43:48.924324 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:43:48.925174 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:43:48.925192 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:43:48.927197 (+ 2005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:43:48.927201 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:43:48.929661 (+ 2460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:43:48.929674 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:43:48.931777 (+ 2103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:43:48.931787 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:43:48.932059 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:43:48.932199 (+ 140us) client_negotiation.cc:769] Sending connection context
0926 03:43:48.932339 (+ 140us) client_negotiation.cc:240] Negotiation successful
0926 03:43:48.932580 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":440,"mutex_wait_us":392,"thread_start_us":293,"threads_started":1}
I20260926 03:43:48.933655 1827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:48.909725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58734 (local address 127.0.194.194:44819)
0926 03:43:48.922182 (+ 12457us) server_negotiation.cc:206] Beginning negotiation
0926 03:43:48.922196 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 03:43:48.924183 (+ 1987us) server_negotiation.cc:407] Connection header received
0926 03:43:48.924287 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:43:48.924294 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:43:48.924524 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:43:48.924715 (+ 191us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:43:48.925565 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:48.926858 (+ 1293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:48.930143 (+ 3285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:48.931522 (+ 1379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:48.931646 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:43:48.931845 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0926 03:43:48.933186 (+ 1341us) server_negotiation.cc:299] Negotiation successful
0926 03:43:48.933380 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12230,"thread_start_us":12501,"threads_started":1}
I20260926 03:43:48.933713 1826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:48.909438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.194:44819 (local address 127.0.0.1:58734)
0926 03:43:48.923891 (+ 14453us) negotiation.cc:107] Waiting for socket to connect
0926 03:43:48.923911 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 03:43:48.924035 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:43:48.924711 (+ 676us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:43:48.924719 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:43:48.924744 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:43:48.925377 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:43:48.925392 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:43:48.926987 (+ 1595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:43:48.926993 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:43:48.929976 (+ 2983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:43:48.929990 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:43:48.931817 (+ 1827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:43:48.931823 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:43:48.932062 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:43:48.932199 (+ 137us) client_negotiation.cc:769] Sending connection context
0926 03:43:48.932339 (+ 140us) client_negotiation.cc:240] Negotiation successful
0926 03:43:48.932580 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14330,"mutex_wait_us":303,"thread_start_us":14571,"threads_started":1}
I20260926 03:43:48.937722 1613 tablet_service.cc:1467] Processing CreateTablet for tablet 5c79ab4071ff4781a789c1af3f073c70 (DEFAULT_TABLE table=test-table [id=5f164b867d2e42f78a620105c9d9e43d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:43:48.937722 1479 tablet_service.cc:1467] Processing CreateTablet for tablet 5c79ab4071ff4781a789c1af3f073c70 (DEFAULT_TABLE table=test-table [id=5f164b867d2e42f78a620105c9d9e43d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:43:48.939314 1479 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c79ab4071ff4781a789c1af3f073c70. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:43:48.939357 1613 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c79ab4071ff4781a789c1af3f073c70. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:43:48.939910 1828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:48.909819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55748 (local address 127.0.194.195:39539)
0926 03:43:48.923013 (+ 13194us) server_negotiation.cc:206] Beginning negotiation
0926 03:43:48.923027 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 03:43:48.923073 (+ 46us) server_negotiation.cc:407] Connection header received
0926 03:43:48.923206 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:43:48.923219 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:43:48.923619 (+ 400us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:43:48.923859 (+ 240us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:43:48.925287 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:48.927023 (+ 1736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:48.931942 (+ 4919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:48.934264 (+ 2322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:48.934416 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:43:48.938599 (+ 4183us) server_negotiation.cc:1035] Waiting for connection context
0926 03:43:48.939504 (+ 905us) server_negotiation.cc:299] Negotiation successful
0926 03:43:48.939657 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12982,"thread_start_us":12790,"threads_started":1}
I20260926 03:43:48.940132 1825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:48.909076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.195:39539 (local address 127.0.0.1:55748)
0926 03:43:48.921183 (+ 12107us) negotiation.cc:107] Waiting for socket to connect
0926 03:43:48.921217 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0926 03:43:48.921432 (+ 215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:43:48.924168 (+ 2736us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:43:48.924178 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:43:48.924220 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:43:48.924944 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:43:48.924961 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:43:48.927164 (+ 2203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:43:48.927172 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:43:48.929976 (+ 2804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:43:48.929990 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:43:48.934463 (+ 4473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:43:48.934469 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:43:48.938670 (+ 4201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:43:48.938713 (+ 43us) client_negotiation.cc:769] Sending connection context
0926 03:43:48.938822 (+ 109us) client_negotiation.cc:240] Negotiation successful
0926 03:43:48.938971 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11896,"mutex_wait_us":314,"thread_start_us":12476,"threads_started":1}
I20260926 03:43:48.943325 1746 tablet_service.cc:1467] Processing CreateTablet for tablet 5c79ab4071ff4781a789c1af3f073c70 (DEFAULT_TABLE table=test-table [id=5f164b867d2e42f78a620105c9d9e43d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:43:48.945369 1746 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c79ab4071ff4781a789c1af3f073c70. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:43:48.954358 1830 tablet_bootstrap.cc:492] T 5c79ab4071ff4781a789c1af3f073c70 P 330f33cc31b74e9baedebff5da38e79f: Bootstrap starting.
I20260926 03:43:48.955165 1829 tablet_bootstrap.cc:492] T 5c79ab4071ff4781a789c1af3f073c70 P 31de5182459c47ada0a3ee8972924d61: Bootstrap starting.
I20260926 03:43:48.957319 1831 tablet_bootstrap.cc:492] T 5c79ab4071ff4781a789c1af3f073c70 P 47b865711c9b4d0485bf6dd46131bd40: Bootstrap starting.
I20260926 03:43:48.960105 1829 tablet_bootstrap.cc:654] T 5c79ab4071ff4781a789c1af3f073c70 P 31de5182459c47ada0a3ee8972924d61: Neither blocks nor log segments found. Creating new log.
I20260926 03:43:48.960392 1831 tablet_bootstrap.cc:654] T 5c79ab4071ff4781a789c1af3f073c70 P 47b865711c9b4d0485bf6dd46131bd40: Neither blocks nor log segments found. Creating new log.
I20260926 03:43:48.961090 1830 tablet_bootstrap.cc:654] T 5c79ab4071ff4781a789c1af3f073c70 P 330f33cc31b74e9baedebff5da38e79f: Neither blocks nor log segments found. Creating new log.
I20260926 03:43:48.961650 1831 log.cc:824] T 5c79ab4071ff4781a789c1af3f073c70 P 47b865711c9b4d0485bf6dd46131bd40: Log is configured to *not* fsync() on all Append() calls
I20260926 03:43:48.961963 1829 log.cc:824] T 5c79ab4071ff4781a789c1af3f073c70 P 31de5182459c47ada0a3ee8972924d61: Log is configured to *not* fsync() on all Append() calls
I20260926 03:43:48.962307 1830 log.cc:824] T 5c79ab4071ff4781a789c1af3f073c70 P 330f33cc31b74e9baedebff5da38e79f: Log is configured to *not* fsync() on all Append() calls
I20260926 03:43:48.964803 1829 tablet_bootstrap.cc:492] T 5c79ab4071ff4781a789c1af3f073c70 P 31de5182459c47ada0a3ee8972924d61: No bootstrap required, opened a new log
I20260926 03:43:48.965065 1830 tablet_bootstrap.cc:492] T 5c79ab4071ff4781a789c1af3f073c70 P 330f33cc31b74e9baedebff5da38e79f: No bootstrap required, opened a new log
I20260926 03:43:48.965083 1829 ts_tablet_manager.cc:1397] T 5c79ab4071ff4781a789c1af3f073c70 P 31de5182459c47ada0a3ee8972924d61: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260926 03:43:48.965344 1830 ts_tablet_manager.cc:1397] T 5c79ab4071ff4781a789c1af3f073c70 P 330f33cc31b74e9baedebff5da38e79f: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.005s
I20260926 03:43:48.968611 1831 tablet_bootstrap.cc:492] T 5c79ab4071ff4781a789c1af3f073c70 P 47b865711c9b4d0485bf6dd46131bd40: No bootstrap required, opened a new log
I20260926 03:43:48.968892 1831 ts_tablet_manager.cc:1397] T 5c79ab4071ff4781a789c1af3f073c70 P 47b865711c9b4d0485bf6dd46131bd40: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.006s
I20260926 03:43:48.973285 1830 raft_consensus.cc:348] T 5c79ab4071ff4781a789c1af3f073c70 P 330f33cc31b74e9baedebff5da38e79f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31de5182459c47ada0a3ee8972924d61" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 45885 } } peers { permanent_uuid: "330f33cc31b74e9baedebff5da38e79f" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 44819 } } peers { permanent_uuid: "47b865711c9b4d0485bf6dd46131bd40" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 39539 } }
I20260926 03:43:48.973685 1830 raft_consensus.cc:374] T 5c79ab4071ff4781a789c1af3f073c70 P 330f33cc31b74e9baedebff5da38e79f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:43:48.973795 1830 raft_consensus.cc:729] T 5c79ab4071ff4781a789c1af3f073c70 P 330f33cc31b74e9baedebff5da38e79f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 330f33cc31b74e9baedebff5da38e79f, State: Initialized, Role: FOLLOWER
I20260926 03:43:48.974467 1830 consensus_queue.cc:260] T 5c79ab4071ff4781a789c1af3f073c70 P 330f33cc31b74e9baedebff5da38e79f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31de5182459c47ada0a3ee8972924d61" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 45885 } } peers { permanent_uuid: "330f33cc31b74e9baedebff5da38e79f" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 44819 } } peers { permanent_uuid: "47b865711c9b4d0485bf6dd46131bd40" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 39539 } }
I20260926 03:43:48.975616 1678 heartbeater.cc:503] Master 127.0.194.254:40529 was elected leader, sending a full tablet report...
I20260926 03:43:48.975507 1829 raft_consensus.cc:348] T 5c79ab4071ff4781a789c1af3f073c70 P 31de5182459c47ada0a3ee8972924d61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31de5182459c47ada0a3ee8972924d61" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 45885 } } peers { permanent_uuid: "330f33cc31b74e9baedebff5da38e79f" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 44819 } } peers { permanent_uuid: "47b865711c9b4d0485bf6dd46131bd40" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 39539 } }
I20260926 03:43:48.975980 1829 raft_consensus.cc:374] T 5c79ab4071ff4781a789c1af3f073c70 P 31de5182459c47ada0a3ee8972924d61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:43:48.976076 1829 raft_consensus.cc:729] T 5c79ab4071ff4781a789c1af3f073c70 P 31de5182459c47ada0a3ee8972924d61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31de5182459c47ada0a3ee8972924d61, State: Initialized, Role: FOLLOWER
I20260926 03:43:48.976663 1829 consensus_queue.cc:260] T 5c79ab4071ff4781a789c1af3f073c70 P 31de5182459c47ada0a3ee8972924d61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31de5182459c47ada0a3ee8972924d61" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 45885 } } peers { permanent_uuid: "330f33cc31b74e9baedebff5da38e79f" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 44819 } } peers { permanent_uuid: "47b865711c9b4d0485bf6dd46131bd40" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 39539 } }
I20260926 03:43:48.977732 1830 ts_tablet_manager.cc:1428] T 5c79ab4071ff4781a789c1af3f073c70 P 330f33cc31b74e9baedebff5da38e79f: Time spent starting tablet: real 0.012s user 0.007s sys 0.005s
I20260926 03:43:48.981890 1831 raft_consensus.cc:348] T 5c79ab4071ff4781a789c1af3f073c70 P 47b865711c9b4d0485bf6dd46131bd40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31de5182459c47ada0a3ee8972924d61" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 45885 } } peers { permanent_uuid: "330f33cc31b74e9baedebff5da38e79f" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 44819 } } peers { permanent_uuid: "47b865711c9b4d0485bf6dd46131bd40" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 39539 } }
I20260926 03:43:48.982486 1831 raft_consensus.cc:374] T 5c79ab4071ff4781a789c1af3f073c70 P 47b865711c9b4d0485bf6dd46131bd40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:43:48.982606 1831 raft_consensus.cc:729] T 5c79ab4071ff4781a789c1af3f073c70 P 47b865711c9b4d0485bf6dd46131bd40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47b865711c9b4d0485bf6dd46131bd40, State: Initialized, Role: FOLLOWER
I20260926 03:43:48.983420 1831 consensus_queue.cc:260] T 5c79ab4071ff4781a789c1af3f073c70 P 47b865711c9b4d0485bf6dd46131bd40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31de5182459c47ada0a3ee8972924d61" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 45885 } } peers { permanent_uuid: "330f33cc31b74e9baedebff5da38e79f" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 44819 } } peers { permanent_uuid: "47b865711c9b4d0485bf6dd46131bd40" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 39539 } }
I20260926 03:43:48.984634 1811 heartbeater.cc:503] Master 127.0.194.254:40529 was elected leader, sending a full tablet report...
I20260926 03:43:48.985576 1829 ts_tablet_manager.cc:1428] T 5c79ab4071ff4781a789c1af3f073c70 P 31de5182459c47ada0a3ee8972924d61: Time spent starting tablet: real 0.020s user 0.016s sys 0.005s
I20260926 03:43:48.986619 1831 ts_tablet_manager.cc:1428] T 5c79ab4071ff4781a789c1af3f073c70 P 47b865711c9b4d0485bf6dd46131bd40: Time spent starting tablet: real 0.018s user 0.014s sys 0.000s
W20260926 03:43:49.065528 1545 tablet.cc:2369] T 5c79ab4071ff4781a789c1af3f073c70 P 31de5182459c47ada0a3ee8972924d61: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 03:43:49.070465 1679 tablet.cc:2369] T 5c79ab4071ff4781a789c1af3f073c70 P 330f33cc31b74e9baedebff5da38e79f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 03:43:49.084770 1812 tablet.cc:2369] T 5c79ab4071ff4781a789c1af3f073c70 P 47b865711c9b4d0485bf6dd46131bd40: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 03:43:49.184127 1837 raft_consensus.cc:482] T 5c79ab4071ff4781a789c1af3f073c70 P 47b865711c9b4d0485bf6dd46131bd40 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 03:43:49.184433 1837 raft_consensus.cc:504] T 5c79ab4071ff4781a789c1af3f073c70 P 47b865711c9b4d0485bf6dd46131bd40 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31de5182459c47ada0a3ee8972924d61" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 45885 } } peers { permanent_uuid: "330f33cc31b74e9baedebff5da38e79f" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 44819 } } peers { permanent_uuid: "47b865711c9b4d0485bf6dd46131bd40" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 39539 } }
I20260926 03:43:49.185415 1836 raft_consensus.cc:482] T 5c79ab4071ff4781a789c1af3f073c70 P 31de5182459c47ada0a3ee8972924d61 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 03:43:49.185688 1836 raft_consensus.cc:504] T 5c79ab4071ff4781a789c1af3f073c70 P 31de5182459c47ada0a3ee8972924d61 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31de5182459c47ada0a3ee8972924d61" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 45885 } } peers { permanent_uuid: "330f33cc31b74e9baedebff5da38e79f" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 44819 } } peers { permanent_uuid: "47b865711c9b4d0485bf6dd46131bd40" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 39539 } }
I20260926 03:43:49.186311 1837 leader_election.cc:288] T 5c79ab4071ff4781a789c1af3f073c70 P 47b865711c9b4d0485bf6dd46131bd40 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31de5182459c47ada0a3ee8972924d61 (127.0.194.193:45885), 330f33cc31b74e9baedebff5da38e79f (127.0.194.194:44819)
I20260926 03:43:49.194168 1836 leader_election.cc:288] T 5c79ab4071ff4781a789c1af3f073c70 P 31de5182459c47ada0a3ee8972924d61 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 330f33cc31b74e9baedebff5da38e79f (127.0.194.194:44819), 47b865711c9b4d0485bf6dd46131bd40 (127.0.194.195:39539)
I20260926 03:43:49.196548 1813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:49.186668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.193:45885 (local address 127.0.194.195:53665)
0926 03:43:49.186874 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0926 03:43:49.186894 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 03:43:49.187041 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:43:49.187537 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:43:49.187543 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:43:49.187576 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:43:49.187839 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:43:49.187850 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:43:49.189675 (+ 1825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:43:49.189684 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:43:49.192469 (+ 2785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:43:49.192489 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:43:49.195465 (+ 2976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:43:49.195474 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:43:49.195766 (+ 292us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:43:49.195819 (+ 53us) client_negotiation.cc:769] Sending connection context
0926 03:43:49.195926 (+ 107us) client_negotiation.cc:240] Negotiation successful
0926 03:43:49.196120 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39,"mutex_wait_us":56}
I20260926 03:43:49.197425 1824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:49.186844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.195:53665 (local address 127.0.194.193:45885)
0926 03:43:49.187010 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0926 03:43:49.187034 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0926 03:43:49.187060 (+ 26us) server_negotiation.cc:407] Connection header received
0926 03:43:49.187170 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:43:49.187175 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:43:49.187258 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:43:49.187369 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:43:49.187995 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:49.189445 (+ 1450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:49.192733 (+ 3288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:49.195053 (+ 2320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:49.195260 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:43:49.195372 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0926 03:43:49.196884 (+ 1512us) server_negotiation.cc:299] Negotiation successful
0926 03:43:49.197074 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 03:43:49.200109 1499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c79ab4071ff4781a789c1af3f073c70" candidate_uuid: "47b865711c9b4d0485bf6dd46131bd40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31de5182459c47ada0a3ee8972924d61" is_pre_election: true
I20260926 03:43:49.200623 1499 raft_consensus.cc:2454] T 5c79ab4071ff4781a789c1af3f073c70 P 31de5182459c47ada0a3ee8972924d61 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 47b865711c9b4d0485bf6dd46131bd40 in term 0.
I20260926 03:43:49.201558 1700 leader_election.cc:302] T 5c79ab4071ff4781a789c1af3f073c70 P 47b865711c9b4d0485bf6dd46131bd40 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31de5182459c47ada0a3ee8972924d61, 47b865711c9b4d0485bf6dd46131bd40; no voters:
I20260926 03:43:49.202021 1837 raft_consensus.cc:2781] T 5c79ab4071ff4781a789c1af3f073c70 P 47b865711c9b4d0485bf6dd46131bd40 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 03:43:49.202208 1837 raft_consensus.cc:482] T 5c79ab4071ff4781a789c1af3f073c70 P 47b865711c9b4d0485bf6dd46131bd40 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 03:43:49.202345 1837 raft_consensus.cc:3037] T 5c79ab4071ff4781a789c1af3f073c70 P 47b865711c9b4d0485bf6dd46131bd40 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:43:49.202638 1838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:49.187112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.194:44819 (local address 127.0.194.195:33587)
0926 03:43:49.188080 (+ 968us) negotiation.cc:107] Waiting for socket to connect
0926 03:43:49.188097 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 03:43:49.188275 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:43:49.188862 (+ 587us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:43:49.188869 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:43:49.188904 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:43:49.189349 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:43:49.189363 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:43:49.191298 (+ 1935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:43:49.191307 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:43:49.199755 (+ 8448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:43:49.199770 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:43:49.201620 (+ 1850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:43:49.201626 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:43:49.201805 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:43:49.201838 (+ 33us) client_negotiation.cc:769] Sending connection context
0926 03:43:49.201933 (+ 95us) client_negotiation.cc:240] Negotiation successful
0926 03:43:49.202076 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":828,"mutex_wait_us":2,"thread_start_us":634,"threads_started":1}
I20260926 03:43:49.203719 1827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:49.187394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.195:33587 (local address 127.0.194.194:44819)
0926 03:43:49.187583 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0926 03:43:49.187589 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:43:49.188166 (+ 577us) server_negotiation.cc:407] Connection header received
0926 03:43:49.188442 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:43:49.188448 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:43:49.188528 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:43:49.188597 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:43:49.189556 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:49.191042 (+ 1486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:49.199957 (+ 8915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:49.201402 (+ 1445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:49.201537 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:43:49.201586 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0926 03:43:49.203319 (+ 1733us) server_negotiation.cc:299] Negotiation successful
0926 03:43:49.203461 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 03:43:49.204710 1633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c79ab4071ff4781a789c1af3f073c70" candidate_uuid: "47b865711c9b4d0485bf6dd46131bd40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "330f33cc31b74e9baedebff5da38e79f" is_pre_election: true
I20260926 03:43:49.205308 1633 raft_consensus.cc:2454] T 5c79ab4071ff4781a789c1af3f073c70 P 330f33cc31b74e9baedebff5da38e79f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 47b865711c9b4d0485bf6dd46131bd40 in term 0.
I20260926 03:43:49.206094 1837 raft_consensus.cc:504] T 5c79ab4071ff4781a789c1af3f073c70 P 47b865711c9b4d0485bf6dd46131bd40 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31de5182459c47ada0a3ee8972924d61" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 45885 } } peers { permanent_uuid: "330f33cc31b74e9baedebff5da38e79f" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 44819 } } peers { permanent_uuid: "47b865711c9b4d0485bf6dd46131bd40" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 39539 } }
I20260926 03:43:49.207696 1499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c79ab4071ff4781a789c1af3f073c70" candidate_uuid: "47b865711c9b4d0485bf6dd46131bd40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31de5182459c47ada0a3ee8972924d61"
I20260926 03:43:49.207922 1499 raft_consensus.cc:3037] T 5c79ab4071ff4781a789c1af3f073c70 P 31de5182459c47ada0a3ee8972924d61 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:43:49.208307 1841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:49.188340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.194:44819 (local address 127.0.194.193:50911)
0926 03:43:49.199348 (+ 11008us) negotiation.cc:107] Waiting for socket to connect
0926 03:43:49.199378 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0926 03:43:49.199547 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:43:49.200719 (+ 1172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:43:49.200725 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:43:49.200764 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:43:49.202055 (+ 1291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:43:49.202069 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:43:49.203521 (+ 1452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:43:49.203528 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:43:49.205734 (+ 2206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:43:49.205748 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:43:49.207401 (+ 1653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:43:49.207408 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:43:49.207581 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:43:49.207620 (+ 39us) client_negotiation.cc:769] Sending connection context
0926 03:43:49.207674 (+ 54us) client_negotiation.cc:240] Negotiation successful
0926 03:43:49.207772 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10840,"thread_start_us":12606,"threads_started":1}
I20260926 03:43:49.207235 1837 leader_election.cc:288] T 5c79ab4071ff4781a789c1af3f073c70 P 47b865711c9b4d0485bf6dd46131bd40 [CANDIDATE]: Term 1 election: Requested vote from peers 31de5182459c47ada0a3ee8972924d61 (127.0.194.193:45885), 330f33cc31b74e9baedebff5da38e79f (127.0.194.194:44819)
I20260926 03:43:49.208678 1840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:49.188353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.193:50911 (local address 127.0.194.194:44819)
0926 03:43:49.200327 (+ 11974us) server_negotiation.cc:206] Beginning negotiation
0926 03:43:49.200335 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:43:49.200365 (+ 30us) server_negotiation.cc:407] Connection header received
0926 03:43:49.200450 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:43:49.200456 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:43:49.200555 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:43:49.200932 (+ 377us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:43:49.202232 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:49.203351 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:49.205931 (+ 2580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:49.207164 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:49.208107 (+ 943us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:43:49.208175 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 03:43:49.208225 (+ 50us) server_negotiation.cc:299] Negotiation successful
0926 03:43:49.208377 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11767,"thread_start_us":12824,"threads_started":1}
I20260926 03:43:49.209362 1828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:49.194723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.193:53711 (local address 127.0.194.195:39539)
0926 03:43:49.194997 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0926 03:43:49.195003 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:43:49.200933 (+ 5930us) server_negotiation.cc:407] Connection header received
0926 03:43:49.202307 (+ 1374us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:43:49.202314 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:43:49.202397 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:43:49.202484 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:43:49.203262 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:49.204458 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:49.206828 (+ 2370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:49.208185 (+ 1357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:49.208351 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:43:49.208444 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0926 03:43:49.209125 (+ 681us) server_negotiation.cc:299] Negotiation successful
0926 03:43:49.209214 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 03:43:49.209547 1839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:49.194520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.195:39539 (local address 127.0.194.193:53711)
0926 03:43:49.200834 (+ 6314us) negotiation.cc:107] Waiting for socket to connect
0926 03:43:49.200858 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 03:43:49.202118 (+ 1260us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:43:49.202678 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:43:49.202682 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:43:49.202708 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:43:49.203077 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:43:49.203088 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:43:49.204640 (+ 1552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:43:49.204646 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:43:49.206673 (+ 2027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:43:49.206688 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:43:49.208764 (+ 2076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:43:49.208770 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:43:49.208942 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:43:49.208977 (+ 35us) client_negotiation.cc:769] Sending connection context
0926 03:43:49.209075 (+ 98us) client_negotiation.cc:240] Negotiation successful
0926 03:43:49.209223 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6145,"thread_start_us":4426,"threads_started":1}
I20260926 03:43:49.210791 1499 raft_consensus.cc:2454] T 5c79ab4071ff4781a789c1af3f073c70 P 31de5182459c47ada0a3ee8972924d61 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 47b865711c9b4d0485bf6dd46131bd40 in term 1.
I20260926 03:43:49.211061 1633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c79ab4071ff4781a789c1af3f073c70" candidate_uuid: "47b865711c9b4d0485bf6dd46131bd40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "330f33cc31b74e9baedebff5da38e79f"
I20260926 03:43:49.211252 1700 leader_election.cc:302] T 5c79ab4071ff4781a789c1af3f073c70 P 47b865711c9b4d0485bf6dd46131bd40 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31de5182459c47ada0a3ee8972924d61, 47b865711c9b4d0485bf6dd46131bd40; no voters:
I20260926 03:43:49.211272 1633 raft_consensus.cc:3037] T 5c79ab4071ff4781a789c1af3f073c70 P 330f33cc31b74e9baedebff5da38e79f [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:43:49.211596 1837 raft_consensus.cc:2781] T 5c79ab4071ff4781a789c1af3f073c70 P 47b865711c9b4d0485bf6dd46131bd40 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:43:49.211884 1632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c79ab4071ff4781a789c1af3f073c70" candidate_uuid: "31de5182459c47ada0a3ee8972924d61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "330f33cc31b74e9baedebff5da38e79f" is_pre_election: true
I20260926 03:43:49.211948 1766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c79ab4071ff4781a789c1af3f073c70" candidate_uuid: "31de5182459c47ada0a3ee8972924d61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47b865711c9b4d0485bf6dd46131bd40" is_pre_election: true
I20260926 03:43:49.213640 1837 raft_consensus.cc:686] T 5c79ab4071ff4781a789c1af3f073c70 P 47b865711c9b4d0485bf6dd46131bd40 [term 1 LEADER]: Becoming Leader. State: Replica: 47b865711c9b4d0485bf6dd46131bd40, State: Running, Role: LEADER
I20260926 03:43:49.214165 1633 raft_consensus.cc:2454] T 5c79ab4071ff4781a789c1af3f073c70 P 330f33cc31b74e9baedebff5da38e79f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 47b865711c9b4d0485bf6dd46131bd40 in term 1.
I20260926 03:43:49.214363 1837 consensus_queue.cc:237] T 5c79ab4071ff4781a789c1af3f073c70 P 47b865711c9b4d0485bf6dd46131bd40 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31de5182459c47ada0a3ee8972924d61" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 45885 } } peers { permanent_uuid: "330f33cc31b74e9baedebff5da38e79f" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 44819 } } peers { permanent_uuid: "47b865711c9b4d0485bf6dd46131bd40" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 39539 } }
I20260926 03:43:49.218282 1435 leader_election.cc:302] T 5c79ab4071ff4781a789c1af3f073c70 P 31de5182459c47ada0a3ee8972924d61 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 31de5182459c47ada0a3ee8972924d61; no voters: 330f33cc31b74e9baedebff5da38e79f, 47b865711c9b4d0485bf6dd46131bd40
I20260926 03:43:49.218676 1836 raft_consensus.cc:2726] T 5c79ab4071ff4781a789c1af3f073c70 P 31de5182459c47ada0a3ee8972924d61 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 03:43:49.219776 1355 catalog_manager.cc:5495] T 5c79ab4071ff4781a789c1af3f073c70 P 47b865711c9b4d0485bf6dd46131bd40 reported cstate change: term changed from 0 to 1, leader changed from <none> to 47b865711c9b4d0485bf6dd46131bd40 (127.0.194.195). New cstate: current_term: 1 leader_uuid: "47b865711c9b4d0485bf6dd46131bd40" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31de5182459c47ada0a3ee8972924d61" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 45885 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "330f33cc31b74e9baedebff5da38e79f" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 44819 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "47b865711c9b4d0485bf6dd46131bd40" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 39539 } health_report { overall_health: HEALTHY } } }
Sep 26 03:43:49 dist-test-slave-n2th krb5kdc[1310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394226, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.194.193@KRBTEST.COM
I20260926 03:43:49.304268 1824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:49.289394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32782 (local address 127.0.194.193:45885)
0926 03:43:49.289624 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0926 03:43:49.289634 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 03:43:49.290066 (+ 432us) server_negotiation.cc:407] Connection header received
0926 03:43:49.290209 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:43:49.290213 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:43:49.290282 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:43:49.290370 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:43:49.292004 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:49.292715 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:49.293969 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:49.294363 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:49.294456 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:43:49.298826 (+ 4370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:43:49.298841 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:43:49.298857 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:43:49.298893 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:43:49.301644 (+ 2751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:43:49.302292 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:43:49.302300 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:43:49.302302 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:43:49.302424 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:43:49.302848 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:43:49.302851 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:43:49.302853 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:43:49.303192 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:43:49.303420 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0926 03:43:49.303714 (+ 294us) server_negotiation.cc:299] Negotiation successful
0926 03:43:49.303883 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
Sep 26 03:43:49 dist-test-slave-n2th krb5kdc[1310](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394229, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 03:43:49 dist-test-slave-n2th krb5kdc[1310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394229, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.194.193@KRBTEST.COM
I20260926 03:43:49.351325 1824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:49.336987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32786 (local address 127.0.194.193:45885)
0926 03:43:49.337220 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0926 03:43:49.337228 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:43:49.337597 (+ 369us) server_negotiation.cc:407] Connection header received
0926 03:43:49.337770 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:43:49.337774 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:43:49.337839 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:43:49.337930 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:43:49.339435 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:49.340181 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:49.341517 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:49.341948 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:49.342056 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:43:49.346178 (+ 4122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:43:49.346183 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:43:49.346187 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:43:49.346222 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:43:49.348837 (+ 2615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:43:49.349481 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:43:49.349485 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:43:49.349487 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:43:49.349637 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:43:49.350012 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:43:49.350015 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:43:49.350017 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:43:49.350273 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:43:49.350466 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0926 03:43:49.350804 (+ 338us) server_negotiation.cc:299] Negotiation successful
0926 03:43:49.350973 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
Sep 26 03:43:49 dist-test-slave-n2th krb5kdc[1310](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394229, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 03:43:49 dist-test-slave-n2th krb5kdc[1310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394229, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.194.193@KRBTEST.COM
I20260926 03:43:49.397212 1824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:49.383314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32790 (local address 127.0.194.193:45885)
0926 03:43:49.383570 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0926 03:43:49.383579 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:43:49.383898 (+ 319us) server_negotiation.cc:407] Connection header received
0926 03:43:49.384122 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:43:49.384126 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:43:49.384186 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:43:49.384295 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:43:49.385598 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:49.386324 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:49.387639 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:49.388073 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:49.388208 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:43:49.392001 (+ 3793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:43:49.392006 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:43:49.392009 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:43:49.392044 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:43:49.394452 (+ 2408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:43:49.395222 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:43:49.395229 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:43:49.395233 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:43:49.395361 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:43:49.395736 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:43:49.395746 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:43:49.395749 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:43:49.396044 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:43:49.396311 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0926 03:43:49.396635 (+ 324us) server_negotiation.cc:299] Negotiation successful
0926 03:43:49.396865 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
W20260926 03:43:49.399086 1459 tablet_service.cc:2908] Rejecting scan request for tablet 5c79ab4071ff4781a789c1af3f073c70: Uninitialized: clean time has not yet been initialized
I20260926 03:43:49.401981 779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu pid 1416
I20260926 03:43:49.414251 1824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:49.402932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32792 (local address 127.0.194.193:45885)
0926 03:43:49.403098 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0926 03:43:49.403102 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:43:49.403572 (+ 470us) server_negotiation.cc:407] Connection header received
0926 03:43:49.403716 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:43:49.403719 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:43:49.403802 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:43:49.403885 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:43:49.405018 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:49.405675 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:49.406938 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:49.407431 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:49.407535 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:43:49.409748 (+ 2213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:43:49.409753 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:43:49.409756 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:43:49.409782 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:43:49.412155 (+ 2373us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:43:49.412789 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:43:49.412793 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:43:49.412795 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:43:49.412903 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:43:49.413194 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:43:49.413197 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:43:49.413199 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:43:49.413438 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:43:49.413628 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0926 03:43:49.413879 (+ 251us) server_negotiation.cc:299] Negotiation successful
0926 03:43:49.414031 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 03:43:49.414994 1539 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:43:49.507500 779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu with pid 1416
I20260926 03:43:49.524821 779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu pid 1550
Sep 26 03:43:49 dist-test-slave-n2th krb5kdc[1310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394229, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.194.194@KRBTEST.COM
I20260926 03:43:49.539592 1840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:49.525610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58742 (local address 127.0.194.194:44819)
0926 03:43:49.525809 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0926 03:43:49.525814 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:43:49.525833 (+ 19us) server_negotiation.cc:407] Connection header received
0926 03:43:49.525914 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:43:49.525918 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:43:49.525987 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:43:49.526066 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:43:49.527299 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:49.528003 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:49.529306 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:49.529682 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:49.529779 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:43:49.534136 (+ 4357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:43:49.534150 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:43:49.534166 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:43:49.534196 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:43:49.536999 (+ 2803us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:43:49.537639 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:43:49.537646 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:43:49.537649 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:43:49.537778 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:43:49.538154 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:43:49.538158 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:43:49.538160 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:43:49.538525 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:43:49.538778 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0926 03:43:49.539145 (+ 367us) server_negotiation.cc:299] Negotiation successful
0926 03:43:49.539309 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260926 03:43:49.540270 1673 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:43:49.627511 779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu with pid 1550
I20260926 03:43:49.644296 779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu pid 1683
I20260926 03:43:49.645108 1838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:49.644580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.193:45885 (local address 127.0.194.195:43019)
0926 03:43:49.644772 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0926 03:43:49.644914 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.193:45885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260926 03:43:49.645454 1700 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.194.193:45885: connect: Connection refused (error 111)
I20260926 03:43:49.646893 1838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:49.646518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.193:45885 (local address 127.0.194.195:37937)
0926 03:43:49.646645 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0926 03:43:49.646715 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.193:45885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260926 03:43:49.647185 1700 consensus_peers.cc:487] T 5c79ab4071ff4781a789c1af3f073c70 P 47b865711c9b4d0485bf6dd46131bd40 -> Peer 31de5182459c47ada0a3ee8972924d61 (127.0.194.193:45885): Couldn't send request to peer 31de5182459c47ada0a3ee8972924d61. Status: Network error: Client connection negotiation failed: client connection to 127.0.194.193:45885: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
Sep 26 03:43:49 dist-test-slave-n2th krb5kdc[1310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394229, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.194.195@KRBTEST.COM
I20260926 03:43:49.659044 1828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:49.645032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55758 (local address 127.0.194.195:39539)
0926 03:43:49.645294 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0926 03:43:49.645300 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:43:49.645321 (+ 21us) server_negotiation.cc:407] Connection header received
0926 03:43:49.645486 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:43:49.645491 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:43:49.645558 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:43:49.645675 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:43:49.647440 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:49.648160 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:49.649349 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:49.649702 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:49.649795 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:43:49.653826 (+ 4031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:43:49.653839 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:43:49.653855 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:43:49.653892 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:43:49.656713 (+ 2821us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:43:49.657345 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:43:49.657352 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:43:49.657354 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:43:49.657486 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:43:49.657790 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:43:49.657793 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:43:49.657795 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:43:49.658132 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:43:49.658354 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0926 03:43:49.658579 (+ 225us) server_negotiation.cc:299] Negotiation successful
0926 03:43:49.658771 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149}
I20260926 03:43:49.659683 1806 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:43:49.752383 1838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:49.751946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.194:44819 (local address 127.0.194.195:50541)
0926 03:43:49.752123 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0926 03:43:49.752220 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.194:44819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260926 03:43:49.752441 779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu with pid 1683
I20260926 03:43:49.769032 779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu pid 1323
I20260926 03:43:49.770255 1385 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:43:49.876361 779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu with pid 1323
2026-09-26T03:43:49Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (4364 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790394221564982-779-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:43:49 dist-test-slave-n2th krb5kdc[1862](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:43:49 dist-test-slave-n2th krb5kdc[1862](info): set up 2 sockets
Sep 26 03:43:49 dist-test-slave-n2th krb5kdc[1862](info): commencing operation
krb5kdc: starting...
W20260926 03:43:50.954695 779 mini_kdc.cc:121] Time spent starting KDC: real 1.036s user 0.004s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790394221564982-779-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790394221564982-779-0/krb5kdc/test-user.keytab.
Sep 26 03:43:51 dist-test-slave-n2th krb5kdc[1862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394231, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:43:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:43:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.194.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.194.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.194.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790394221564982-779-0/krb5kdc/kudu_127.0.194.254.keytab.
Entry for principal kudu/127.0.194.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790394221564982-779-0/krb5kdc/kudu_127.0.194.254.keytab.
I20260926 03:43:51.144205 779 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
/tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790394221564982-779-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790394221564982-779-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790394221564982-779-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790394221564982-779-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.194.254:37875
--webserver_interface=127.0.194.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790394221564982-779-0/krb5kdc/kudu_127.0.194.254.keytab
--principal=kudu/127.0.194.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.194.212:34187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.194.254:37875
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790394221564982-779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790394221564982-779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790394221564982-779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:43:51.464069 1875 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:43:51.464375 1875 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:43:51.464533 1875 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:43:51.471828 1875 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:43:51.471946 1875 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260926 03:43:51.471987 1875 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:43:51.472049 1875 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:43:51.472105 1875 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:43:51.472167 1875 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:43:51.482082 1875 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.194.212:34187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790394221564982-779-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790394221564982-779-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.194.254:37875
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790394221564982-779-0/krb5kdc/kudu_127.0.194.254.keytab
--principal=kudu/127.0.194.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.194.254:37875
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790394221564982-779-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.194.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790394221564982-779-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:43:51.483350 1875 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:43:51.484629 1875 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:43:51.496398 1883 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:43:51.496464 1880 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:43:51.496488 1881 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:43:51.497630 1875 server_base.cc:1034] running on GCE node
I20260926 03:43:51.498466 1875 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:43:51.500003 1875 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:43:51.501235 1875 hybrid_clock.cc:648] HybridClock initialized: now 1790394231501176 us; error 35 us; skew 500 ppm
I20260926 03:43:51.501590 1875 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:43:51 dist-test-slave-n2th krb5kdc[1862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394231, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:43:51.504823 1875 init.cc:377] Logged in from keytab as kudu/127.0.194.254@KRBTEST.COM (short username kudu)
I20260926 03:43:51.506331 1875 webserver.cc:460] Webserver started at http://127.0.194.254:37585/ using document root <none> and password file <none>
I20260926 03:43:51.507112 1875 fs_manager.cc:359] Metadata directory not provided
I20260926 03:43:51.507215 1875 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:43:51.507488 1875 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:43:51.510548 1875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790394221564982-779-0/minicluster-data/master-0/data/instance:
uuid: "5908aa01446f46239684277b300e638a"
format_stamp: "Formatted at 2026-09-26 03:43:51 on dist-test-slave-n2th"
server_key: "36733451610966ce2026c3ef67ad261b"
server_key_iv: "e6292fb7e978d0c0c083aa404c4a5de1"
server_key_version: "encryptionkey@0"
I20260926 03:43:51.511492 1875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790394221564982-779-0/minicluster-data/master-0/wal/instance:
uuid: "5908aa01446f46239684277b300e638a"
format_stamp: "Formatted at 2026-09-26 03:43:51 on dist-test-slave-n2th"
server_key: "36733451610966ce2026c3ef67ad261b"
server_key_iv: "e6292fb7e978d0c0c083aa404c4a5de1"
server_key_version: "encryptionkey@0"
I20260926 03:43:51.516839 1875 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.002s
I20260926 03:43:51.520545 1890 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:43:51.521929 1875 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 03:43:51.522114 1875 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790394221564982-779-0/minicluster-data/master-0/data,/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790394221564982-779-0/minicluster-data/master-0/wal
uuid: "5908aa01446f46239684277b300e638a"
format_stamp: "Formatted at 2026-09-26 03:43:51 on dist-test-slave-n2th"
server_key: "36733451610966ce2026c3ef67ad261b"
server_key_iv: "e6292fb7e978d0c0c083aa404c4a5de1"
server_key_version: "encryptionkey@0"
I20260926 03:43:51.522352 1875 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790394221564982-779-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790394221564982-779-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790394221564982-779-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:43:51.547396 1875 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:43:51.551671 1875 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:43:51.551949 1875 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:43:51.573920 1875 rpc_server.cc:307] RPC server started. Bound to: 127.0.194.254:37875
I20260926 03:43:51.573927 1951 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.194.254:37875 every 8 connection(s)
I20260926 03:43:51.575385 1875 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790394221564982-779-0/minicluster-data/master-0/data/info.pb
I20260926 03:43:51.579345 1952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:43:51.582684 779 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu as pid 1875
I20260926 03:43:51.582943 779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790394221564982-779-0/minicluster-data/master-0/wal/instance
I20260926 03:43:51.583775 779 external_mini_cluster.cc:1428] Setting key 1c591e7b4b234ce40a0ce9c54d870c31
I20260926 03:43:51.592890 1952 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5908aa01446f46239684277b300e638a: Bootstrap starting.
I20260926 03:43:51.596923 1952 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5908aa01446f46239684277b300e638a: Neither blocks nor log segments found. Creating new log.
I20260926 03:43:51.598366 1952 log.cc:824] T 00000000000000000000000000000000 P 5908aa01446f46239684277b300e638a: Log is configured to *not* fsync() on all Append() calls
Sep 26 03:43:51 dist-test-slave-n2th krb5kdc[1862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394231, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.194.254@KRBTEST.COM
I20260926 03:43:51.601389 1952 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5908aa01446f46239684277b300e638a: No bootstrap required, opened a new log
I20260926 03:43:51.608075 1955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:51.586013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53006 (local address 127.0.194.254:37875)
0926 03:43:51.586862 (+ 849us) server_negotiation.cc:206] Beginning negotiation
0926 03:43:51.586874 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 03:43:51.586916 (+ 42us) server_negotiation.cc:407] Connection header received
0926 03:43:51.588611 (+ 1695us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:43:51.588640 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:43:51.589079 (+ 439us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:43:51.589436 (+ 357us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:43:51.591271 (+ 1835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:51.592731 (+ 1460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:51.594460 (+ 1729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:51.595266 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:51.595406 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:43:51.600557 (+ 5151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:43:51.600568 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:43:51.600584 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:43:51.600622 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:43:51.603853 (+ 3231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:43:51.604603 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:43:51.604614 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:43:51.604632 (+ 18us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:43:51.604802 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:43:51.605196 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:43:51.605207 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:43:51.605210 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:43:51.605735 (+ 525us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:43:51.606057 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0926 03:43:51.606509 (+ 452us) server_negotiation.cc:299] Negotiation successful
0926 03:43:51.606982 (+ 473us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":572,"thread_start_us":367,"threads_started":1}
I20260926 03:43:51.609161 1952 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5908aa01446f46239684277b300e638a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5908aa01446f46239684277b300e638a" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 37875 } }
I20260926 03:43:51.609530 1952 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5908aa01446f46239684277b300e638a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:43:51.609658 1952 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5908aa01446f46239684277b300e638a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5908aa01446f46239684277b300e638a, State: Initialized, Role: FOLLOWER
I20260926 03:43:51.610237 1952 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5908aa01446f46239684277b300e638a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5908aa01446f46239684277b300e638a" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 37875 } }
I20260926 03:43:51.610473 1952 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5908aa01446f46239684277b300e638a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:43:51.610599 1952 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5908aa01446f46239684277b300e638a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:43:51.610797 1952 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5908aa01446f46239684277b300e638a [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:43:51.613062 1952 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5908aa01446f46239684277b300e638a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5908aa01446f46239684277b300e638a" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 37875 } }
I20260926 03:43:51.613570 1952 leader_election.cc:302] T 00000000000000000000000000000000 P 5908aa01446f46239684277b300e638a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5908aa01446f46239684277b300e638a; no voters:
I20260926 03:43:51.614137 1952 leader_election.cc:288] T 00000000000000000000000000000000 P 5908aa01446f46239684277b300e638a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:43:51.614276 1957 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5908aa01446f46239684277b300e638a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:43:51.614972 1957 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5908aa01446f46239684277b300e638a [term 1 LEADER]: Becoming Leader. State: Replica: 5908aa01446f46239684277b300e638a, State: Running, Role: LEADER
I20260926 03:43:51.615610 1957 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5908aa01446f46239684277b300e638a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5908aa01446f46239684277b300e638a" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 37875 } }
I20260926 03:43:51.617034 1952 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5908aa01446f46239684277b300e638a [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:43:51.618696 1958 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5908aa01446f46239684277b300e638a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5908aa01446f46239684277b300e638a. Latest consensus state: current_term: 1 leader_uuid: "5908aa01446f46239684277b300e638a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5908aa01446f46239684277b300e638a" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 37875 } } }
I20260926 03:43:51.619113 1958 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5908aa01446f46239684277b300e638a [sys.catalog]: This master's current role is: LEADER
I20260926 03:43:51.619565 1959 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5908aa01446f46239684277b300e638a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5908aa01446f46239684277b300e638a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5908aa01446f46239684277b300e638a" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 37875 } } }
I20260926 03:43:51.619875 1959 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5908aa01446f46239684277b300e638a [sys.catalog]: This master's current role is: LEADER
I20260926 03:43:51.624019 1964 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:43:51.629626 1964 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:43:51.639043 1964 catalog_manager.cc:1347] Generated new cluster ID: e5c6d88f262d4386b85d61eb369e7c75
I20260926 03:43:51.639194 1964 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:43:51.647488 1964 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:43:51.649093 1964 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:43:51.659518 1964 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5908aa01446f46239684277b300e638a: Generated new TSK 0
I20260926 03:43:51.660657 1964 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.194.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.194.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.194.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790394221564982-779-0/krb5kdc/kudu_127.0.194.193.keytab.
Entry for principal kudu/127.0.194.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790394221564982-779-0/krb5kdc/kudu_127.0.194.193.keytab.
I20260926 03:43:51.717255 779 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
/tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790394221564982-779-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790394221564982-779-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790394221564982-779-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790394221564982-779-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.194.193:0
--local_ip_for_outbound_sockets=127.0.194.193
--webserver_interface=127.0.194.193
--webserver_port=0
--tserver_master_addrs=127.0.194.254:37875
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790394221564982-779-0/krb5kdc/kudu_127.0.194.193.keytab
--principal=kudu/127.0.194.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.194.212:34187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790394221564982-779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790394221564982-779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790394221564982-779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:43:52.047473 1978 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:43:52.047773 1978 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:43:52.047933 1978 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:43:52.055351 1978 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:43:52.055466 1978 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:43:52.055606 1978 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 03:43:52.055723 1978 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.194.193
I20260926 03:43:52.065699 1978 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.194.212:34187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790394221564982-779-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790394221564982-779-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790394221564982-779-0/krb5kdc/kudu_127.0.194.193.keytab
--principal=kudu/127.0.194.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.194.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790394221564982-779-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.194.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.194.254:37875
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.194.193
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790394221564982-779-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:43:52.067044 1978 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:43:52.068372 1978 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:43:52.081560 1984 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:43:52.081518 1986 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:43:52.082017 1983 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:43:52.082142 1978 server_base.cc:1034] running on GCE node
I20260926 03:43:52.082888 1978 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:43:52.083871 1978 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:43:52.085143 1978 hybrid_clock.cc:648] HybridClock initialized: now 1790394232085080 us; error 39 us; skew 500 ppm
I20260926 03:43:52.085480 1978 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:43:52 dist-test-slave-n2th krb5kdc[1862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394232, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:43:52.088673 1978 init.cc:377] Logged in from keytab as kudu/127.0.194.193@KRBTEST.COM (short username kudu)
I20260926 03:43:52.090036 1978 webserver.cc:460] Webserver started at http://127.0.194.193:33397/ using document root <none> and password file <none>
I20260926 03:43:52.090806 1978 fs_manager.cc:359] Metadata directory not provided
I20260926 03:43:52.090927 1978 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:43:52.091298 1978 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:43:52.094254 1978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790394221564982-779-0/minicluster-data/ts-0/data/instance:
uuid: "df904b94fdc14a05b7d78665d955f86f"
format_stamp: "Formatted at 2026-09-26 03:43:52 on dist-test-slave-n2th"
server_key: "0c257d2cd74486d719db262bbbc0ee9d"
server_key_iv: "5c311f35195617e32de4f361daff5894"
server_key_version: "encryptionkey@0"
I20260926 03:43:52.095053 1978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790394221564982-779-0/minicluster-data/ts-0/wal/instance:
uuid: "df904b94fdc14a05b7d78665d955f86f"
format_stamp: "Formatted at 2026-09-26 03:43:52 on dist-test-slave-n2th"
server_key: "0c257d2cd74486d719db262bbbc0ee9d"
server_key_iv: "5c311f35195617e32de4f361daff5894"
server_key_version: "encryptionkey@0"
I20260926 03:43:52.100059 1978 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260926 03:43:52.103565 1993 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:43:52.104928 1978 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 03:43:52.105115 1978 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790394221564982-779-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790394221564982-779-0/minicluster-data/ts-0/wal
uuid: "df904b94fdc14a05b7d78665d955f86f"
format_stamp: "Formatted at 2026-09-26 03:43:52 on dist-test-slave-n2th"
server_key: "0c257d2cd74486d719db262bbbc0ee9d"
server_key_iv: "5c311f35195617e32de4f361daff5894"
server_key_version: "encryptionkey@0"
I20260926 03:43:52.105341 1978 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790394221564982-779-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790394221564982-779-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790394221564982-779-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:43:52.126343 1978 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:43:52.130519 1978 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:43:52.130825 1978 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:43:52.133492 1978 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:43:52.133579 1978 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:43:52.133744 1978 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:43:52.133803 1978 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 03:43:52 dist-test-slave-n2th krb5kdc[1862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394232, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.193@KRBTEST.COM for kudu/127.0.194.254@KRBTEST.COM
I20260926 03:43:52.158075 2002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:52.134365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.193:45799 (local address 127.0.194.254:37875)
0926 03:43:52.134825 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0926 03:43:52.134832 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:43:52.136241 (+ 1409us) server_negotiation.cc:407] Connection header received
0926 03:43:52.138551 (+ 2310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:43:52.138559 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:43:52.138689 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:43:52.138853 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:43:52.141168 (+ 2315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:52.142222 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:52.144698 (+ 2476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:52.145362 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:52.145503 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:43:52.150804 (+ 5301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:43:52.150812 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:43:52.150819 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:43:52.150868 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:43:52.154338 (+ 3470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:43:52.155478 (+ 1140us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:43:52.155487 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:43:52.155494 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:43:52.155675 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:43:52.156222 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:43:52.156228 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:43:52.156231 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:43:52.156542 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:43:52.156706 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0926 03:43:52.157468 (+ 762us) server_negotiation.cc:299] Negotiation successful
0926 03:43:52.157705 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":214,"threads_started":1}
I20260926 03:43:52.159584 2003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:52.135142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:37875 (local address 127.0.194.193:45799)
0926 03:43:52.136031 (+ 889us) negotiation.cc:107] Waiting for socket to connect
0926 03:43:52.136078 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 03:43:52.138110 (+ 2032us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:43:52.139143 (+ 1033us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:43:52.139152 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:43:52.139963 (+ 811us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:43:52.140972 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:43:52.140993 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:43:52.142402 (+ 1409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:43:52.142413 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:43:52.144529 (+ 2116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:43:52.144542 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:43:52.145622 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:43:52.145634 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:43:52.145873 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:43:52.146934 (+ 1061us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:43:52.146997 (+ 63us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:43:52.150437 (+ 3440us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:43:52.154492 (+ 4055us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:43:52.154506 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:43:52.154524 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:43:52.155295 (+ 771us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:43:52.155817 (+ 522us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:43:52.155823 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:43:52.155830 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:43:52.156061 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:43:52.156687 (+ 626us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:43:52.156696 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:43:52.157099 (+ 403us) client_negotiation.cc:769] Sending connection context
0926 03:43:52.157379 (+ 280us) client_negotiation.cc:240] Negotiation successful
0926 03:43:52.157942 (+ 563us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":612,"thread_start_us":404,"threads_started":1}
I20260926 03:43:52.180861 1978 rpc_server.cc:307] RPC server started. Bound to: 127.0.194.193:33137
I20260926 03:43:52.180899 2113 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.194.193:33137 every 8 connection(s)
I20260926 03:43:52.182391 1978 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790394221564982-779-0/minicluster-data/ts-0/data/info.pb
I20260926 03:43:52.184998 2114 heartbeater.cc:347] Connected to a master server at 127.0.194.254:37875
I20260926 03:43:52.185487 2114 heartbeater.cc:464] Registering TS with master...
I20260926 03:43:52.186199 2114 heartbeater.cc:511] Master 127.0.194.254:37875 requested a full tablet report, sending...
I20260926 03:43:52.187705 2002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:52.166379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.193:43265 (local address 127.0.194.254:37875)
0926 03:43:52.166605 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0926 03:43:52.166615 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 03:43:52.167349 (+ 734us) server_negotiation.cc:407] Connection header received
0926 03:43:52.167584 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:43:52.167588 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:43:52.167651 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:43:52.167737 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:43:52.169200 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:52.169888 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:52.172344 (+ 2456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:52.172841 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:52.172937 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:43:52.181829 (+ 8892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:43:52.181834 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:43:52.181837 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:43:52.181873 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:43:52.184866 (+ 2993us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:43:52.185679 (+ 813us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:43:52.185684 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:43:52.185687 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:43:52.185826 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:43:52.186206 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:43:52.186211 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:43:52.186215 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:43:52.186533 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:43:52.186689 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0926 03:43:52.187244 (+ 555us) server_negotiation.cc:299] Negotiation successful
0926 03:43:52.187457 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 03:43:52.187849 2081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:52.166359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:37875 (local address 127.0.194.193:43265)
0926 03:43:52.167217 (+ 858us) negotiation.cc:107] Waiting for socket to connect
0926 03:43:52.167246 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0926 03:43:52.167425 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:43:52.167804 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:43:52.167812 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:43:52.168551 (+ 739us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:43:52.169026 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:43:52.169043 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:43:52.170014 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:43:52.170022 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:43:52.172170 (+ 2148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:43:52.172192 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:43:52.173000 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:43:52.173008 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:43:52.173192 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:43:52.180339 (+ 7147us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:43:52.180372 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:43:52.181603 (+ 1231us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:43:52.185011 (+ 3408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:43:52.185019 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:43:52.185024 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:43:52.185567 (+ 543us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:43:52.185943 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:43:52.185948 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:43:52.185952 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:43:52.186086 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:43:52.186645 (+ 559us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:43:52.186650 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:43:52.186888 (+ 238us) client_negotiation.cc:769] Sending connection context
0926 03:43:52.187044 (+ 156us) client_negotiation.cc:240] Negotiation successful
0926 03:43:52.187386 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"thread_start_us":398,"threads_started":1}
I20260926 03:43:52.188477 779 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu as pid 1978
I20260926 03:43:52.188587 1907 ts_manager.cc:194] Registered new tserver with Master: df904b94fdc14a05b7d78665d955f86f (127.0.194.193:33137)
I20260926 03:43:52.188697 779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790394221564982-779-0/minicluster-data/ts-0/wal/instance
I20260926 03:43:52.189473 779 external_mini_cluster.cc:1428] Setting key 260f5706fd6eacfd33f10c0191eac4b7
I20260926 03:43:52.190641 1907 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.194.193@KRBTEST.COM'} at 127.0.194.193:45799
WARNING: no policy specified for kudu/127.0.194.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.194.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.194.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790394221564982-779-0/krb5kdc/kudu_127.0.194.194.keytab.
Entry for principal kudu/127.0.194.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790394221564982-779-0/krb5kdc/kudu_127.0.194.194.keytab.
I20260926 03:43:52.231804 779 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
/tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790394221564982-779-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790394221564982-779-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790394221564982-779-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790394221564982-779-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.194.194:0
--local_ip_for_outbound_sockets=127.0.194.194
--webserver_interface=127.0.194.194
--webserver_port=0
--tserver_master_addrs=127.0.194.254:37875
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790394221564982-779-0/krb5kdc/kudu_127.0.194.194.keytab
--principal=kudu/127.0.194.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.194.212:34187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790394221564982-779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790394221564982-779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790394221564982-779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:43:52.555037 2118 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:43:52.555337 2118 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:43:52.555521 2118 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:43:52.562908 2118 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:43:52.563045 2118 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:43:52.563205 2118 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 03:43:52.563331 2118 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.194.194
I20260926 03:43:52.573138 2118 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.194.212:34187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790394221564982-779-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790394221564982-779-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790394221564982-779-0/krb5kdc/kudu_127.0.194.194.keytab
--principal=kudu/127.0.194.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.194.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790394221564982-779-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.194.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.194.254:37875
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.194.194
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790394221564982-779-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:43:52.574504 2118 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:43:52.575855 2118 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:43:52.591377 2123 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:43:52.591377 2126 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:43:52.591318 2124 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:43:52.592331 2118 server_base.cc:1034] running on GCE node
I20260926 03:43:52.592974 2118 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:43:52.593956 2118 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:43:52.595176 2118 hybrid_clock.cc:648] HybridClock initialized: now 1790394232595115 us; error 37 us; skew 500 ppm
I20260926 03:43:52.595449 2118 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:43:52 dist-test-slave-n2th krb5kdc[1862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394232, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:43:52.598591 2118 init.cc:377] Logged in from keytab as kudu/127.0.194.194@KRBTEST.COM (short username kudu)
I20260926 03:43:52.600266 2118 webserver.cc:460] Webserver started at http://127.0.194.194:37821/ using document root <none> and password file <none>
I20260926 03:43:52.600960 2118 fs_manager.cc:359] Metadata directory not provided
I20260926 03:43:52.601049 2118 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:43:52.601311 2118 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:43:52.604373 2118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790394221564982-779-0/minicluster-data/ts-1/data/instance:
uuid: "88b981e00e3e43089b67f936a8e86e51"
format_stamp: "Formatted at 2026-09-26 03:43:52 on dist-test-slave-n2th"
server_key: "0e12eeb89c1571bb13466d5bee02f867"
server_key_iv: "4590e67859f76d073b2b22eb7a865575"
server_key_version: "encryptionkey@0"
I20260926 03:43:52.605137 2118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790394221564982-779-0/minicluster-data/ts-1/wal/instance:
uuid: "88b981e00e3e43089b67f936a8e86e51"
format_stamp: "Formatted at 2026-09-26 03:43:52 on dist-test-slave-n2th"
server_key: "0e12eeb89c1571bb13466d5bee02f867"
server_key_iv: "4590e67859f76d073b2b22eb7a865575"
server_key_version: "encryptionkey@0"
I20260926 03:43:52.610536 2118 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 03:43:52.614168 2133 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:43:52.615659 2118 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 03:43:52.615852 2118 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790394221564982-779-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790394221564982-779-0/minicluster-data/ts-1/wal
uuid: "88b981e00e3e43089b67f936a8e86e51"
format_stamp: "Formatted at 2026-09-26 03:43:52 on dist-test-slave-n2th"
server_key: "0e12eeb89c1571bb13466d5bee02f867"
server_key_iv: "4590e67859f76d073b2b22eb7a865575"
server_key_version: "encryptionkey@0"
I20260926 03:43:52.616088 2118 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790394221564982-779-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790394221564982-779-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790394221564982-779-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:43:52.643674 2118 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:43:52.648658 2118 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:43:52.648960 2118 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:43:52.652060 2118 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:43:52.652182 2118 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:43:52.652349 2118 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:43:52.652433 2118 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 03:43:52 dist-test-slave-n2th krb5kdc[1862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394232, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.194@KRBTEST.COM for kudu/127.0.194.254@KRBTEST.COM
I20260926 03:43:52.679307 2002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:52.652738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.194:40631 (local address 127.0.194.254:37875)
0926 03:43:52.652974 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0926 03:43:52.652983 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:43:52.654524 (+ 1541us) server_negotiation.cc:407] Connection header received
0926 03:43:52.656647 (+ 2123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:43:52.656656 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:43:52.656805 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:43:52.656931 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:43:52.659346 (+ 2415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:52.660747 (+ 1401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:52.663570 (+ 2823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:52.664334 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:52.664483 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:43:52.671067 (+ 6584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:43:52.671079 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:43:52.671084 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:43:52.671130 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:43:52.675340 (+ 4210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:43:52.676672 (+ 1332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:43:52.676679 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:43:52.676684 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:43:52.676850 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:43:52.677415 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:43:52.677421 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:43:52.677425 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:43:52.677741 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:43:52.677900 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0926 03:43:52.678678 (+ 778us) server_negotiation.cc:299] Negotiation successful
0926 03:43:52.678930 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 03:43:52.680648 2142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:52.653346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:37875 (local address 127.0.194.194:40631)
0926 03:43:52.654315 (+ 969us) negotiation.cc:107] Waiting for socket to connect
0926 03:43:52.654369 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0926 03:43:52.656354 (+ 1985us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:43:52.657216 (+ 862us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:43:52.657231 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:43:52.658121 (+ 890us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:43:52.659075 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:43:52.659132 (+ 57us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:43:52.660954 (+ 1822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:43:52.660993 (+ 39us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:43:52.663433 (+ 2440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:43:52.663449 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:43:52.664861 (+ 1412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:43:52.664876 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:43:52.665275 (+ 399us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:43:52.666375 (+ 1100us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:43:52.666418 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:43:52.670655 (+ 4237us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:43:52.675564 (+ 4909us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:43:52.675582 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:43:52.675611 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:43:52.676473 (+ 862us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:43:52.677008 (+ 535us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:43:52.677014 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:43:52.677020 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:43:52.677274 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:43:52.677877 (+ 603us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:43:52.677886 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:43:52.678265 (+ 379us) client_negotiation.cc:769] Sending connection context
0926 03:43:52.678504 (+ 239us) client_negotiation.cc:240] Negotiation successful
0926 03:43:52.679201 (+ 697us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":677,"thread_start_us":382,"threads_started":1}
I20260926 03:43:52.703096 2002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:52.686374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.194:50333 (local address 127.0.194.254:37875)
0926 03:43:52.686606 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0926 03:43:52.686615 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:43:52.687281 (+ 666us) server_negotiation.cc:407] Connection header received
0926 03:43:52.687512 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:43:52.687518 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:43:52.687623 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:43:52.687773 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:43:52.688805 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:52.690302 (+ 1497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:52.692944 (+ 2642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:52.693372 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:52.693474 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:43:52.696300 (+ 2826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:43:52.696304 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:43:52.696308 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:43:52.696341 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:43:52.699556 (+ 3215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:43:52.700766 (+ 1210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:43:52.700771 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:43:52.700776 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:43:52.700966 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:43:52.701430 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:43:52.701435 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:43:52.701439 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:43:52.701922 (+ 483us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:43:52.702080 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0926 03:43:52.702480 (+ 400us) server_negotiation.cc:299] Negotiation successful
0926 03:43:52.702757 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 03:43:52.703226 2220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:52.686387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:37875 (local address 127.0.194.194:50333)
0926 03:43:52.687160 (+ 773us) negotiation.cc:107] Waiting for socket to connect
0926 03:43:52.687177 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 03:43:52.687362 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:43:52.687789 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:43:52.687793 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:43:52.688358 (+ 565us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:43:52.688657 (+ 299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:43:52.688667 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:43:52.690521 (+ 1854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:43:52.690531 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:43:52.692766 (+ 2235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:43:52.692787 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:43:52.693586 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:43:52.693597 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:43:52.693828 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:43:52.694609 (+ 781us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:43:52.694642 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:43:52.696067 (+ 1425us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:43:52.699743 (+ 3676us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:43:52.699752 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:43:52.699760 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:43:52.700566 (+ 806us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:43:52.701126 (+ 560us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:43:52.701137 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:43:52.701142 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:43:52.701293 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:43:52.702044 (+ 751us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:43:52.702050 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:43:52.702236 (+ 186us) client_negotiation.cc:769] Sending connection context
0926 03:43:52.702378 (+ 142us) client_negotiation.cc:240] Negotiation successful
0926 03:43:52.702755 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":611,"thread_start_us":404,"threads_started":1}
I20260926 03:43:52.711436 2118 rpc_server.cc:307] RPC server started. Bound to: 127.0.194.194:46757
I20260926 03:43:52.711480 2252 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.194.194:46757 every 8 connection(s)
I20260926 03:43:52.712847 2118 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790394221564982-779-0/minicluster-data/ts-1/data/info.pb
I20260926 03:43:52.715261 2253 heartbeater.cc:347] Connected to a master server at 127.0.194.254:37875
I20260926 03:43:52.715512 2253 heartbeater.cc:464] Registering TS with master...
I20260926 03:43:52.716202 2253 heartbeater.cc:511] Master 127.0.194.254:37875 requested a full tablet report, sending...
I20260926 03:43:52.717625 1907 ts_manager.cc:194] Registered new tserver with Master: 88b981e00e3e43089b67f936a8e86e51 (127.0.194.194:46757)
I20260926 03:43:52.718554 1907 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.194.194@KRBTEST.COM'} at 127.0.194.194:40631
I20260926 03:43:52.720685 779 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu as pid 2118
I20260926 03:43:52.720875 779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790394221564982-779-0/minicluster-data/ts-1/wal/instance
I20260926 03:43:52.721336 779 external_mini_cluster.cc:1428] Setting key 2438c492b63f5b91396c4771c428d24d
WARNING: no policy specified for kudu/127.0.194.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.194.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.194.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790394221564982-779-0/krb5kdc/kudu_127.0.194.195.keytab.
Entry for principal kudu/127.0.194.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790394221564982-779-0/krb5kdc/kudu_127.0.194.195.keytab.
I20260926 03:43:52.761201 779 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
/tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790394221564982-779-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790394221564982-779-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790394221564982-779-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790394221564982-779-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.194.195:0
--local_ip_for_outbound_sockets=127.0.194.195
--webserver_interface=127.0.194.195
--webserver_port=0
--tserver_master_addrs=127.0.194.254:37875
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790394221564982-779-0/krb5kdc/kudu_127.0.194.195.keytab
--principal=kudu/127.0.194.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.194.212:34187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790394221564982-779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790394221564982-779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790394221564982-779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:43:53.089147 2257 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:43:53.089448 2257 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:43:53.089650 2257 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:43:53.097045 2257 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:43:53.097162 2257 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:43:53.097316 2257 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 03:43:53.097447 2257 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.194.195
I20260926 03:43:53.107465 2257 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.194.212:34187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790394221564982-779-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790394221564982-779-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790394221564982-779-0/krb5kdc/kudu_127.0.194.195.keytab
--principal=kudu/127.0.194.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.194.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790394221564982-779-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.194.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.194.254:37875
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.194.195
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790394221564982-779-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:43:53.108860 2257 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:43:53.110245 2257 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:43:53.123595 2265 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:43:53.123670 2263 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:43:53.123608 2262 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:43:53.124709 2257 server_base.cc:1034] running on GCE node
I20260926 03:43:53.125406 2257 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:43:53.126364 2257 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:43:53.127635 2257 hybrid_clock.cc:648] HybridClock initialized: now 1790394233127569 us; error 37 us; skew 500 ppm
I20260926 03:43:53.127920 2257 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:43:53 dist-test-slave-n2th krb5kdc[1862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394233, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:43:53.131179 2257 init.cc:377] Logged in from keytab as kudu/127.0.194.195@KRBTEST.COM (short username kudu)
I20260926 03:43:53.132634 2257 webserver.cc:460] Webserver started at http://127.0.194.195:34109/ using document root <none> and password file <none>
I20260926 03:43:53.133315 2257 fs_manager.cc:359] Metadata directory not provided
I20260926 03:43:53.133405 2257 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:43:53.133663 2257 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:43:53.136587 2257 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790394221564982-779-0/minicluster-data/ts-2/data/instance:
uuid: "cdbcd498e18240d88a06242e77df5360"
format_stamp: "Formatted at 2026-09-26 03:43:53 on dist-test-slave-n2th"
server_key: "158bdf3680b5757ec420da2bb760f9f7"
server_key_iv: "fff42d325e3315c6399991c16388d814"
server_key_version: "encryptionkey@0"
I20260926 03:43:53.137318 2257 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790394221564982-779-0/minicluster-data/ts-2/wal/instance:
uuid: "cdbcd498e18240d88a06242e77df5360"
format_stamp: "Formatted at 2026-09-26 03:43:53 on dist-test-slave-n2th"
server_key: "158bdf3680b5757ec420da2bb760f9f7"
server_key_iv: "fff42d325e3315c6399991c16388d814"
server_key_version: "encryptionkey@0"
I20260926 03:43:53.142441 2257 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260926 03:43:53.145932 2272 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:43:53.147325 2257 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260926 03:43:53.147507 2257 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790394221564982-779-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790394221564982-779-0/minicluster-data/ts-2/wal
uuid: "cdbcd498e18240d88a06242e77df5360"
format_stamp: "Formatted at 2026-09-26 03:43:53 on dist-test-slave-n2th"
server_key: "158bdf3680b5757ec420da2bb760f9f7"
server_key_iv: "fff42d325e3315c6399991c16388d814"
server_key_version: "encryptionkey@0"
I20260926 03:43:53.147724 2257 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790394221564982-779-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790394221564982-779-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790394221564982-779-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:43:53.165947 2257 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:43:53.170336 2257 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:43:53.170624 2257 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:43:53.173195 2257 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:43:53.173317 2257 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:43:53.173457 2257 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:43:53.173523 2257 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 03:43:53 dist-test-slave-n2th krb5kdc[1862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394233, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.195@KRBTEST.COM for kudu/127.0.194.254@KRBTEST.COM
I20260926 03:43:53.194466 2114 heartbeater.cc:503] Master 127.0.194.254:37875 was elected leader, sending a full tablet report...
I20260926 03:43:53.196463 2002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:53.174095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.195:53757 (local address 127.0.194.254:37875)
0926 03:43:53.174275 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0926 03:43:53.174279 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:43:53.175814 (+ 1535us) server_negotiation.cc:407] Connection header received
0926 03:43:53.177790 (+ 1976us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:43:53.177797 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:43:53.177890 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:43:53.178000 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:43:53.180353 (+ 2353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:53.181468 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:53.184537 (+ 3069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:53.185276 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:53.185410 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:43:53.189560 (+ 4150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:43:53.189567 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:43:53.189572 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:43:53.189613 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:43:53.192443 (+ 2830us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:43:53.193443 (+ 1000us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:43:53.193450 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:43:53.193453 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:43:53.193588 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:43:53.194061 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:43:53.194066 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:43:53.194069 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:43:53.194340 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:43:53.194495 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0926 03:43:53.195819 (+ 1324us) server_negotiation.cc:299] Negotiation successful
0926 03:43:53.196082 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 03:43:53.198344 2281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:53.174789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:37875 (local address 127.0.194.195:53757)
0926 03:43:53.175659 (+ 870us) negotiation.cc:107] Waiting for socket to connect
0926 03:43:53.175695 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 03:43:53.177498 (+ 1803us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:43:53.178356 (+ 858us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:43:53.178370 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:43:53.179288 (+ 918us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:43:53.180131 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:43:53.180154 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:43:53.181653 (+ 1499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:43:53.181657 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:43:53.184343 (+ 2686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:43:53.184361 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:43:53.185537 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:43:53.185546 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:43:53.185772 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:43:53.186495 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:43:53.186521 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:43:53.189306 (+ 2785us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:43:53.192610 (+ 3304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:43:53.192618 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:43:53.192635 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:43:53.193304 (+ 669us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:43:53.193703 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:43:53.193708 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:43:53.193712 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:43:53.193932 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:43:53.194478 (+ 546us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:43:53.194486 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:43:53.195080 (+ 594us) client_negotiation.cc:769] Sending connection context
0926 03:43:53.195478 (+ 398us) client_negotiation.cc:240] Negotiation successful
0926 03:43:53.196221 (+ 743us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":619,"thread_start_us":399,"threads_started":1}
I20260926 03:43:53.219942 2257 rpc_server.cc:307] RPC server started. Bound to: 127.0.194.195:33291
I20260926 03:43:53.219981 2391 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.194.195:33291 every 8 connection(s)
I20260926 03:43:53.221546 2257 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790394221564982-779-0/minicluster-data/ts-2/data/info.pb
I20260926 03:43:53.223779 2392 heartbeater.cc:347] Connected to a master server at 127.0.194.254:37875
I20260926 03:43:53.224017 2392 heartbeater.cc:464] Registering TS with master...
I20260926 03:43:53.224834 2357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:53.206944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:37875 (local address 127.0.194.195:41393)
0926 03:43:53.207730 (+ 786us) negotiation.cc:107] Waiting for socket to connect
0926 03:43:53.207756 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0926 03:43:53.208005 (+ 249us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:43:53.208749 (+ 744us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:43:53.208760 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:43:53.209439 (+ 679us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:43:53.210201 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:43:53.210220 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:43:53.211516 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:43:53.211527 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:43:53.213594 (+ 2067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:43:53.213610 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:43:53.215101 (+ 1491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:43:53.215110 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:43:53.215354 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:43:53.216216 (+ 862us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:43:53.216244 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:43:53.217515 (+ 1271us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:43:53.221547 (+ 4032us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:43:53.221551 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:43:53.221554 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:43:53.221989 (+ 435us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:43:53.222530 (+ 541us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:43:53.222538 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:43:53.222542 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:43:53.222683 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:43:53.223624 (+ 941us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:43:53.223629 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:43:53.223836 (+ 207us) client_negotiation.cc:769] Sending connection context
0926 03:43:53.223981 (+ 145us) client_negotiation.cc:240] Negotiation successful
0926 03:43:53.224337 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":589,"thread_start_us":649,"threads_started":1}
I20260926 03:43:53.224897 2392 heartbeater.cc:511] Master 127.0.194.254:37875 requested a full tablet report, sending...
I20260926 03:43:53.225749 2002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:53.206715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.195:41393 (local address 127.0.194.254:37875)
0926 03:43:53.206960 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0926 03:43:53.206967 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:43:53.207862 (+ 895us) server_negotiation.cc:407] Connection header received
0926 03:43:53.208217 (+ 355us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:43:53.208224 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:43:53.208317 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:43:53.208403 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:43:53.210426 (+ 2023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:53.211301 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:53.213803 (+ 2502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:53.214578 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:53.214697 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:43:53.217707 (+ 3010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:43:53.217712 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:43:53.217715 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:43:53.217745 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:43:53.221413 (+ 3668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:43:53.222176 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:43:53.222181 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:43:53.222185 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:43:53.222357 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:43:53.223053 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:43:53.223060 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:43:53.223064 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:43:53.223497 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:43:53.223648 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0926 03:43:53.225147 (+ 1499us) server_negotiation.cc:299] Negotiation successful
0926 03:43:53.225402 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 03:43:53.226398 1905 ts_manager.cc:194] Registered new tserver with Master: cdbcd498e18240d88a06242e77df5360 (127.0.194.195:33291)
I20260926 03:43:53.227571 1905 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.194.195@KRBTEST.COM'} at 127.0.194.195:53757
I20260926 03:43:53.229411 779 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu as pid 2257
I20260926 03:43:53.229607 779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790394221564982-779-0/minicluster-data/ts-2/wal/instance
I20260926 03:43:53.230120 779 external_mini_cluster.cc:1428] Setting key 3fa1f51caa9f5f54ee0af0019d4ad3dd
I20260926 03:43:53.232780 779 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 03:43:53 dist-test-slave-n2th krb5kdc[1862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394233, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 03:43:53 dist-test-slave-n2th krb5kdc[1862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394233, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.194.254@KRBTEST.COM
I20260926 03:43:53.280448 2002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:53.265135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53016 (local address 127.0.194.254:37875)
0926 03:43:53.265400 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0926 03:43:53.265408 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:43:53.265743 (+ 335us) server_negotiation.cc:407] Connection header received
0926 03:43:53.265967 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:43:53.265976 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:43:53.266049 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:43:53.266161 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:43:53.267781 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:53.268568 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:53.269873 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:53.270307 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:53.270457 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:43:53.274719 (+ 4262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:43:53.274724 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:43:53.274775 (+ 51us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:43:53.274827 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:43:53.277610 (+ 2783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:43:53.278358 (+ 748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:43:53.278364 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:43:53.278367 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:43:53.278503 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:43:53.278972 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:43:53.278977 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:43:53.278980 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:43:53.279271 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:43:53.279533 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0926 03:43:53.279837 (+ 304us) server_negotiation.cc:299] Negotiation successful
0926 03:43:53.280064 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260926 03:43:53.285338 1905 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53016:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 03:43:53.292639 1905 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 03:43:53.330227 2405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:53.306202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.193:33137 (local address 127.0.0.1:36392)
0926 03:43:53.319969 (+ 13767us) negotiation.cc:107] Waiting for socket to connect
0926 03:43:53.319993 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 03:43:53.320126 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:43:53.322293 (+ 2167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:43:53.322301 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:43:53.322341 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:43:53.322989 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:43:53.323015 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:43:53.325161 (+ 2146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:43:53.325167 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:43:53.327547 (+ 2380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:43:53.327561 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:43:53.329138 (+ 1577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:43:53.329146 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:43:53.329331 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:43:53.329398 (+ 67us) client_negotiation.cc:769] Sending connection context
0926 03:43:53.329542 (+ 144us) client_negotiation.cc:240] Negotiation successful
0926 03:43:53.329742 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13503,"mutex_wait_us":27,"thread_start_us":16483,"threads_started":1}
I20260926 03:43:53.330438 2406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:53.306219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36772 (local address 127.0.194.195:33291)
0926 03:43:53.321694 (+ 15475us) server_negotiation.cc:206] Beginning negotiation
0926 03:43:53.321701 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:43:53.321727 (+ 26us) server_negotiation.cc:407] Connection header received
0926 03:43:53.321799 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:43:53.321804 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:43:53.321971 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:43:53.323229 (+ 1258us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:43:53.323610 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:53.325239 (+ 1629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:53.327963 (+ 2724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:53.329317 (+ 1354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:53.329444 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:43:53.329670 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0926 03:43:53.329915 (+ 245us) server_negotiation.cc:299] Negotiation successful
0926 03:43:53.330086 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15341,"thread_start_us":15927,"threads_started":1}
I20260926 03:43:53.331053 2409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:53.306994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36392 (local address 127.0.194.193:33137)
0926 03:43:53.321448 (+ 14454us) server_negotiation.cc:206] Beginning negotiation
0926 03:43:53.321461 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 03:43:53.321501 (+ 40us) server_negotiation.cc:407] Connection header received
0926 03:43:53.321642 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:43:53.321655 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:43:53.321999 (+ 344us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:43:53.322292 (+ 293us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:43:53.323249 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:53.325020 (+ 1771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:53.327752 (+ 2732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:53.328936 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:53.329083 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:43:53.329295 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0926 03:43:53.330599 (+ 1304us) server_negotiation.cc:299] Negotiation successful
0926 03:43:53.330793 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14278,"thread_start_us":14154,"threads_started":1}
I20260926 03:43:53.330655 2408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:53.306009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.195:33291 (local address 127.0.0.1:36772)
0926 03:43:53.318697 (+ 12688us) negotiation.cc:107] Waiting for socket to connect
0926 03:43:53.318843 (+ 146us) client_negotiation.cc:174] Beginning negotiation
0926 03:43:53.319137 (+ 294us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:43:53.322376 (+ 3239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:43:53.322383 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:43:53.322408 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:43:53.323009 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:43:53.323017 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:43:53.325366 (+ 2349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:43:53.325374 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:43:53.327773 (+ 2399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:43:53.327787 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:43:53.329530 (+ 1743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:43:53.329536 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:43:53.329705 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:43:53.329735 (+ 30us) client_negotiation.cc:769] Sending connection context
0926 03:43:53.329820 (+ 85us) client_negotiation.cc:240] Negotiation successful
0926 03:43:53.329958 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11318,"mutex_wait_us":24,"thread_start_us":13493,"threads_started":1}
I20260926 03:43:53.331512 2410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:53.307409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44226 (local address 127.0.194.194:46757)
0926 03:43:53.321007 (+ 13598us) server_negotiation.cc:206] Beginning negotiation
0926 03:43:53.321020 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 03:43:53.323156 (+ 2136us) server_negotiation.cc:407] Connection header received
0926 03:43:53.323477 (+ 321us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:43:53.323486 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:43:53.323728 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:43:53.323883 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:43:53.324669 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:53.326078 (+ 1409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:53.328751 (+ 2673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:53.330131 (+ 1380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:53.330275 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:43:53.330464 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0926 03:43:53.331050 (+ 586us) server_negotiation.cc:299] Negotiation successful
0926 03:43:53.331228 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13433,"thread_start_us":13313,"threads_started":1}
I20260926 03:43:53.331760 2407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:53.306843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.194:46757 (local address 127.0.0.1:44226)
0926 03:43:53.323038 (+ 16195us) negotiation.cc:107] Waiting for socket to connect
0926 03:43:53.323058 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 03:43:53.323260 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:43:53.324130 (+ 870us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:43:53.324134 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:43:53.324159 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:43:53.324498 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:43:53.324507 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:43:53.326220 (+ 1713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:43:53.326225 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:43:53.328588 (+ 2363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:43:53.328601 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:43:53.330710 (+ 2109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:43:53.330719 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:43:53.330875 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:43:53.330897 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 03:43:53.330977 (+ 80us) client_negotiation.cc:240] Negotiation successful
0926 03:43:53.331059 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16033,"thread_start_us":10875,"threads_started":1}
I20260926 03:43:53.334982 2183 tablet_service.cc:1467] Processing CreateTablet for tablet d6f253aa2cc740ba9d837af8fcebc0a6 (DEFAULT_TABLE table=test-table [id=43645a3081c74664b982f75f346c5f09]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:43:53.335314 2044 tablet_service.cc:1467] Processing CreateTablet for tablet d6f253aa2cc740ba9d837af8fcebc0a6 (DEFAULT_TABLE table=test-table [id=43645a3081c74664b982f75f346c5f09]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:43:53.335314 2322 tablet_service.cc:1467] Processing CreateTablet for tablet d6f253aa2cc740ba9d837af8fcebc0a6 (DEFAULT_TABLE table=test-table [id=43645a3081c74664b982f75f346c5f09]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:43:53.336284 2183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6f253aa2cc740ba9d837af8fcebc0a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:43:53.336617 2044 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6f253aa2cc740ba9d837af8fcebc0a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:43:53.336630 2322 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6f253aa2cc740ba9d837af8fcebc0a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:43:53.347760 2411 tablet_bootstrap.cc:492] T d6f253aa2cc740ba9d837af8fcebc0a6 P 88b981e00e3e43089b67f936a8e86e51: Bootstrap starting.
I20260926 03:43:53.351123 2411 tablet_bootstrap.cc:654] T d6f253aa2cc740ba9d837af8fcebc0a6 P 88b981e00e3e43089b67f936a8e86e51: Neither blocks nor log segments found. Creating new log.
I20260926 03:43:53.352552 2411 log.cc:824] T d6f253aa2cc740ba9d837af8fcebc0a6 P 88b981e00e3e43089b67f936a8e86e51: Log is configured to *not* fsync() on all Append() calls
I20260926 03:43:53.355316 2411 tablet_bootstrap.cc:492] T d6f253aa2cc740ba9d837af8fcebc0a6 P 88b981e00e3e43089b67f936a8e86e51: No bootstrap required, opened a new log
I20260926 03:43:53.355389 2413 tablet_bootstrap.cc:492] T d6f253aa2cc740ba9d837af8fcebc0a6 P df904b94fdc14a05b7d78665d955f86f: Bootstrap starting.
I20260926 03:43:53.355623 2411 ts_tablet_manager.cc:1397] T d6f253aa2cc740ba9d837af8fcebc0a6 P 88b981e00e3e43089b67f936a8e86e51: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260926 03:43:53.358232 2413 tablet_bootstrap.cc:654] T d6f253aa2cc740ba9d837af8fcebc0a6 P df904b94fdc14a05b7d78665d955f86f: Neither blocks nor log segments found. Creating new log.
I20260926 03:43:53.358549 2412 tablet_bootstrap.cc:492] T d6f253aa2cc740ba9d837af8fcebc0a6 P cdbcd498e18240d88a06242e77df5360: Bootstrap starting.
I20260926 03:43:53.359500 2413 log.cc:824] T d6f253aa2cc740ba9d837af8fcebc0a6 P df904b94fdc14a05b7d78665d955f86f: Log is configured to *not* fsync() on all Append() calls
I20260926 03:43:53.362051 2412 tablet_bootstrap.cc:654] T d6f253aa2cc740ba9d837af8fcebc0a6 P cdbcd498e18240d88a06242e77df5360: Neither blocks nor log segments found. Creating new log.
I20260926 03:43:53.363296 2412 log.cc:824] T d6f253aa2cc740ba9d837af8fcebc0a6 P cdbcd498e18240d88a06242e77df5360: Log is configured to *not* fsync() on all Append() calls
I20260926 03:43:53.365082 2411 raft_consensus.cc:348] T d6f253aa2cc740ba9d837af8fcebc0a6 P 88b981e00e3e43089b67f936a8e86e51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdbcd498e18240d88a06242e77df5360" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 33291 } } peers { permanent_uuid: "df904b94fdc14a05b7d78665d955f86f" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 33137 } } peers { permanent_uuid: "88b981e00e3e43089b67f936a8e86e51" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 46757 } }
I20260926 03:43:53.365463 2411 raft_consensus.cc:374] T d6f253aa2cc740ba9d837af8fcebc0a6 P 88b981e00e3e43089b67f936a8e86e51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:43:53.365568 2411 raft_consensus.cc:729] T d6f253aa2cc740ba9d837af8fcebc0a6 P 88b981e00e3e43089b67f936a8e86e51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88b981e00e3e43089b67f936a8e86e51, State: Initialized, Role: FOLLOWER
I20260926 03:43:53.366173 2411 consensus_queue.cc:260] T d6f253aa2cc740ba9d837af8fcebc0a6 P 88b981e00e3e43089b67f936a8e86e51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdbcd498e18240d88a06242e77df5360" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 33291 } } peers { permanent_uuid: "df904b94fdc14a05b7d78665d955f86f" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 33137 } } peers { permanent_uuid: "88b981e00e3e43089b67f936a8e86e51" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 46757 } }
I20260926 03:43:53.366456 2412 tablet_bootstrap.cc:492] T d6f253aa2cc740ba9d837af8fcebc0a6 P cdbcd498e18240d88a06242e77df5360: No bootstrap required, opened a new log
I20260926 03:43:53.366451 2413 tablet_bootstrap.cc:492] T d6f253aa2cc740ba9d837af8fcebc0a6 P df904b94fdc14a05b7d78665d955f86f: No bootstrap required, opened a new log
I20260926 03:43:53.366756 2413 ts_tablet_manager.cc:1397] T d6f253aa2cc740ba9d837af8fcebc0a6 P df904b94fdc14a05b7d78665d955f86f: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.005s
I20260926 03:43:53.366758 2412 ts_tablet_manager.cc:1397] T d6f253aa2cc740ba9d837af8fcebc0a6 P cdbcd498e18240d88a06242e77df5360: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.000s
I20260926 03:43:53.367332 2253 heartbeater.cc:503] Master 127.0.194.254:37875 was elected leader, sending a full tablet report...
I20260926 03:43:53.367939 2411 ts_tablet_manager.cc:1428] T d6f253aa2cc740ba9d837af8fcebc0a6 P 88b981e00e3e43089b67f936a8e86e51: Time spent starting tablet: real 0.012s user 0.009s sys 0.003s
I20260926 03:43:53.377513 2413 raft_consensus.cc:348] T d6f253aa2cc740ba9d837af8fcebc0a6 P df904b94fdc14a05b7d78665d955f86f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdbcd498e18240d88a06242e77df5360" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 33291 } } peers { permanent_uuid: "df904b94fdc14a05b7d78665d955f86f" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 33137 } } peers { permanent_uuid: "88b981e00e3e43089b67f936a8e86e51" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 46757 } }
I20260926 03:43:53.377513 2412 raft_consensus.cc:348] T d6f253aa2cc740ba9d837af8fcebc0a6 P cdbcd498e18240d88a06242e77df5360 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdbcd498e18240d88a06242e77df5360" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 33291 } } peers { permanent_uuid: "df904b94fdc14a05b7d78665d955f86f" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 33137 } } peers { permanent_uuid: "88b981e00e3e43089b67f936a8e86e51" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 46757 } }
I20260926 03:43:53.378010 2412 raft_consensus.cc:374] T d6f253aa2cc740ba9d837af8fcebc0a6 P cdbcd498e18240d88a06242e77df5360 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:43:53.378011 2413 raft_consensus.cc:374] T d6f253aa2cc740ba9d837af8fcebc0a6 P df904b94fdc14a05b7d78665d955f86f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:43:53.378118 2413 raft_consensus.cc:729] T d6f253aa2cc740ba9d837af8fcebc0a6 P df904b94fdc14a05b7d78665d955f86f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df904b94fdc14a05b7d78665d955f86f, State: Initialized, Role: FOLLOWER
I20260926 03:43:53.378123 2412 raft_consensus.cc:729] T d6f253aa2cc740ba9d837af8fcebc0a6 P cdbcd498e18240d88a06242e77df5360 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cdbcd498e18240d88a06242e77df5360, State: Initialized, Role: FOLLOWER
I20260926 03:43:53.378808 2412 consensus_queue.cc:260] T d6f253aa2cc740ba9d837af8fcebc0a6 P cdbcd498e18240d88a06242e77df5360 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdbcd498e18240d88a06242e77df5360" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 33291 } } peers { permanent_uuid: "df904b94fdc14a05b7d78665d955f86f" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 33137 } } peers { permanent_uuid: "88b981e00e3e43089b67f936a8e86e51" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 46757 } }
I20260926 03:43:53.378808 2413 consensus_queue.cc:260] T d6f253aa2cc740ba9d837af8fcebc0a6 P df904b94fdc14a05b7d78665d955f86f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdbcd498e18240d88a06242e77df5360" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 33291 } } peers { permanent_uuid: "df904b94fdc14a05b7d78665d955f86f" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 33137 } } peers { permanent_uuid: "88b981e00e3e43089b67f936a8e86e51" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 46757 } }
I20260926 03:43:53.380429 2392 heartbeater.cc:503] Master 127.0.194.254:37875 was elected leader, sending a full tablet report...
I20260926 03:43:53.380839 2413 ts_tablet_manager.cc:1428] T d6f253aa2cc740ba9d837af8fcebc0a6 P df904b94fdc14a05b7d78665d955f86f: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260926 03:43:53.380882 2412 ts_tablet_manager.cc:1428] T d6f253aa2cc740ba9d837af8fcebc0a6 P cdbcd498e18240d88a06242e77df5360: Time spent starting tablet: real 0.014s user 0.012s sys 0.001s
I20260926 03:43:53.434193 2418 raft_consensus.cc:482] T d6f253aa2cc740ba9d837af8fcebc0a6 P cdbcd498e18240d88a06242e77df5360 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 03:43:53.434494 2418 raft_consensus.cc:504] T d6f253aa2cc740ba9d837af8fcebc0a6 P cdbcd498e18240d88a06242e77df5360 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdbcd498e18240d88a06242e77df5360" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 33291 } } peers { permanent_uuid: "df904b94fdc14a05b7d78665d955f86f" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 33137 } } peers { permanent_uuid: "88b981e00e3e43089b67f936a8e86e51" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 46757 } }
W20260926 03:43:53.434937 2115 tablet.cc:2369] T d6f253aa2cc740ba9d837af8fcebc0a6 P df904b94fdc14a05b7d78665d955f86f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 03:43:53.436170 2418 leader_election.cc:288] T d6f253aa2cc740ba9d837af8fcebc0a6 P cdbcd498e18240d88a06242e77df5360 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df904b94fdc14a05b7d78665d955f86f (127.0.194.193:33137), 88b981e00e3e43089b67f936a8e86e51 (127.0.194.194:46757)
I20260926 03:43:53.442773 2410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:53.436694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.195:50645 (local address 127.0.194.194:46757)
0926 03:43:53.436879 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0926 03:43:53.436886 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:43:53.436905 (+ 19us) server_negotiation.cc:407] Connection header received
0926 03:43:53.437108 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:43:53.437112 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:43:53.437174 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:43:53.437273 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:43:53.437709 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:53.438687 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:53.440564 (+ 1877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:53.441742 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:53.441873 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:43:53.441934 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 03:43:53.442363 (+ 429us) server_negotiation.cc:299] Negotiation successful
0926 03:43:53.442505 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 03:43:53.442716 2281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:53.436447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.194:46757 (local address 127.0.194.195:50645)
0926 03:43:53.436797 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0926 03:43:53.436815 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 03:43:53.436946 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:43:53.437289 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:43:53.437294 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:43:53.437319 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:43:53.437560 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:43:53.437571 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:43:53.438842 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:43:53.438847 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:43:53.440378 (+ 1531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:43:53.440390 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:43:53.441898 (+ 1508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:43:53.441904 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:43:53.442138 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:43:53.442176 (+ 38us) client_negotiation.cc:769] Sending connection context
0926 03:43:53.442315 (+ 139us) client_negotiation.cc:240] Negotiation successful
0926 03:43:53.442456 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"mutex_wait_us":43}
I20260926 03:43:53.443248 2409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:53.436484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.195:52203 (local address 127.0.194.193:33137)
0926 03:43:53.436733 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0926 03:43:53.436740 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:43:53.437465 (+ 725us) server_negotiation.cc:407] Connection header received
0926 03:43:53.437600 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:43:53.437608 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:43:53.437694 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:43:53.437771 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:43:53.438364 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:53.439267 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:53.441282 (+ 2015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:53.442261 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:53.442409 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:43:53.442507 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 03:43:53.442893 (+ 386us) server_negotiation.cc:299] Negotiation successful
0926 03:43:53.443034 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260926 03:43:53.443348 2420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:53.436588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.193:33137 (local address 127.0.194.195:52203)
0926 03:43:53.437366 (+ 778us) negotiation.cc:107] Waiting for socket to connect
0926 03:43:53.437379 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 03:43:53.437484 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:43:53.437821 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:43:53.437826 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:43:53.437850 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:43:53.438205 (+ 355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:43:53.438217 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:43:53.439457 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:43:53.439461 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:43:53.441055 (+ 1594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:43:53.441069 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:43:53.442512 (+ 1443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:43:53.442519 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:43:53.442688 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:43:53.442753 (+ 65us) client_negotiation.cc:769] Sending connection context
0926 03:43:53.442857 (+ 104us) client_negotiation.cc:240] Negotiation successful
0926 03:43:53.443005 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":659,"mutex_wait_us":4,"thread_start_us":471,"threads_started":1}
I20260926 03:43:53.444314 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6f253aa2cc740ba9d837af8fcebc0a6" candidate_uuid: "cdbcd498e18240d88a06242e77df5360" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df904b94fdc14a05b7d78665d955f86f" is_pre_election: true
I20260926 03:43:53.444273 2203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6f253aa2cc740ba9d837af8fcebc0a6" candidate_uuid: "cdbcd498e18240d88a06242e77df5360" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b981e00e3e43089b67f936a8e86e51" is_pre_election: true
I20260926 03:43:53.444823 2064 raft_consensus.cc:2454] T d6f253aa2cc740ba9d837af8fcebc0a6 P df904b94fdc14a05b7d78665d955f86f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cdbcd498e18240d88a06242e77df5360 in term 0.
I20260926 03:43:53.444825 2203 raft_consensus.cc:2454] T d6f253aa2cc740ba9d837af8fcebc0a6 P 88b981e00e3e43089b67f936a8e86e51 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cdbcd498e18240d88a06242e77df5360 in term 0.
I20260926 03:43:53.445508 2275 leader_election.cc:302] T d6f253aa2cc740ba9d837af8fcebc0a6 P cdbcd498e18240d88a06242e77df5360 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cdbcd498e18240d88a06242e77df5360, df904b94fdc14a05b7d78665d955f86f; no voters:
I20260926 03:43:53.445941 2418 raft_consensus.cc:2781] T d6f253aa2cc740ba9d837af8fcebc0a6 P cdbcd498e18240d88a06242e77df5360 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 03:43:53.446085 2418 raft_consensus.cc:482] T d6f253aa2cc740ba9d837af8fcebc0a6 P cdbcd498e18240d88a06242e77df5360 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 03:43:53.446182 2418 raft_consensus.cc:3037] T d6f253aa2cc740ba9d837af8fcebc0a6 P cdbcd498e18240d88a06242e77df5360 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:43:53.448338 2418 raft_consensus.cc:504] T d6f253aa2cc740ba9d837af8fcebc0a6 P cdbcd498e18240d88a06242e77df5360 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdbcd498e18240d88a06242e77df5360" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 33291 } } peers { permanent_uuid: "df904b94fdc14a05b7d78665d955f86f" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 33137 } } peers { permanent_uuid: "88b981e00e3e43089b67f936a8e86e51" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 46757 } }
I20260926 03:43:53.448964 2418 leader_election.cc:288] T d6f253aa2cc740ba9d837af8fcebc0a6 P cdbcd498e18240d88a06242e77df5360 [CANDIDATE]: Term 1 election: Requested vote from peers df904b94fdc14a05b7d78665d955f86f (127.0.194.193:33137), 88b981e00e3e43089b67f936a8e86e51 (127.0.194.194:46757)
I20260926 03:43:53.449337 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6f253aa2cc740ba9d837af8fcebc0a6" candidate_uuid: "cdbcd498e18240d88a06242e77df5360" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df904b94fdc14a05b7d78665d955f86f"
I20260926 03:43:53.449489 2203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6f253aa2cc740ba9d837af8fcebc0a6" candidate_uuid: "cdbcd498e18240d88a06242e77df5360" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b981e00e3e43089b67f936a8e86e51"
I20260926 03:43:53.449533 2064 raft_consensus.cc:3037] T d6f253aa2cc740ba9d837af8fcebc0a6 P df904b94fdc14a05b7d78665d955f86f [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:43:53.449712 2203 raft_consensus.cc:3037] T d6f253aa2cc740ba9d837af8fcebc0a6 P 88b981e00e3e43089b67f936a8e86e51 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:43:53.451903 2064 raft_consensus.cc:2454] T d6f253aa2cc740ba9d837af8fcebc0a6 P df904b94fdc14a05b7d78665d955f86f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cdbcd498e18240d88a06242e77df5360 in term 1.
I20260926 03:43:53.452054 2203 raft_consensus.cc:2454] T d6f253aa2cc740ba9d837af8fcebc0a6 P 88b981e00e3e43089b67f936a8e86e51 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cdbcd498e18240d88a06242e77df5360 in term 1.
I20260926 03:43:53.452308 2275 leader_election.cc:302] T d6f253aa2cc740ba9d837af8fcebc0a6 P cdbcd498e18240d88a06242e77df5360 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cdbcd498e18240d88a06242e77df5360, df904b94fdc14a05b7d78665d955f86f; no voters:
I20260926 03:43:53.452647 2418 raft_consensus.cc:2781] T d6f253aa2cc740ba9d837af8fcebc0a6 P cdbcd498e18240d88a06242e77df5360 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:43:53.453007 2418 raft_consensus.cc:686] T d6f253aa2cc740ba9d837af8fcebc0a6 P cdbcd498e18240d88a06242e77df5360 [term 1 LEADER]: Becoming Leader. State: Replica: cdbcd498e18240d88a06242e77df5360, State: Running, Role: LEADER
I20260926 03:43:53.453456 2418 consensus_queue.cc:237] T d6f253aa2cc740ba9d837af8fcebc0a6 P cdbcd498e18240d88a06242e77df5360 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdbcd498e18240d88a06242e77df5360" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 33291 } } peers { permanent_uuid: "df904b94fdc14a05b7d78665d955f86f" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 33137 } } peers { permanent_uuid: "88b981e00e3e43089b67f936a8e86e51" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 46757 } }
I20260926 03:43:53.458052 1906 catalog_manager.cc:5495] T d6f253aa2cc740ba9d837af8fcebc0a6 P cdbcd498e18240d88a06242e77df5360 reported cstate change: term changed from 0 to 1, leader changed from <none> to cdbcd498e18240d88a06242e77df5360 (127.0.194.195). New cstate: current_term: 1 leader_uuid: "cdbcd498e18240d88a06242e77df5360" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdbcd498e18240d88a06242e77df5360" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 33291 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df904b94fdc14a05b7d78665d955f86f" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 33137 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88b981e00e3e43089b67f936a8e86e51" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 46757 } health_report { overall_health: UNKNOWN } } }
W20260926 03:43:53.465147 2254 tablet.cc:2369] T d6f253aa2cc740ba9d837af8fcebc0a6 P 88b981e00e3e43089b67f936a8e86e51: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 03:43:53.473248 2393 tablet.cc:2369] T d6f253aa2cc740ba9d837af8fcebc0a6 P cdbcd498e18240d88a06242e77df5360: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 03:43:53.491657 2406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:53.486493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36788 (local address 127.0.194.195:33291)
0926 03:43:53.486725 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0926 03:43:53.486769 (+ 44us) server_negotiation.cc:399] Waiting for connection header
0926 03:43:53.486790 (+ 21us) server_negotiation.cc:407] Connection header received
0926 03:43:53.487061 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:43:53.487065 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:43:53.487145 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:43:53.487262 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:43:53.487566 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:53.488249 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:53.489473 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:53.489871 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:53.489972 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:43:53.490388 (+ 416us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:43:53.490964 (+ 576us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:43:53.491066 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 03:43:53.491225 (+ 159us) server_negotiation.cc:299] Negotiation successful
0926 03:43:53.491324 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 03:43:53.499537 2203 raft_consensus.cc:1261] T d6f253aa2cc740ba9d837af8fcebc0a6 P 88b981e00e3e43089b67f936a8e86e51 [term 1 FOLLOWER]: Refusing update from remote peer cdbcd498e18240d88a06242e77df5360: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:43:53.499565 2064 raft_consensus.cc:1261] T d6f253aa2cc740ba9d837af8fcebc0a6 P df904b94fdc14a05b7d78665d955f86f [term 1 FOLLOWER]: Refusing update from remote peer cdbcd498e18240d88a06242e77df5360: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:43:53.500500 2421 consensus_queue.cc:1035] T d6f253aa2cc740ba9d837af8fcebc0a6 P cdbcd498e18240d88a06242e77df5360 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df904b94fdc14a05b7d78665d955f86f" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 33137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:43:53.500908 2418 consensus_queue.cc:1035] T d6f253aa2cc740ba9d837af8fcebc0a6 P cdbcd498e18240d88a06242e77df5360 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b981e00e3e43089b67f936a8e86e51" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 46757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:43:53.515630 2426 mvcc.cc:205] Tried to move back new op lower bound from 7333454780404760576 to 7333454780229287936. Current Snapshot: MvccSnapshot[applied={T|T < 7333454780404760576}]
I20260926 03:43:53.530431 1905 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53016:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:43:53.530704 1905 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53016:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:43:53.534307 1905 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5908aa01446f46239684277b300e638a: Sending DeleteTablet for 3 replicas of tablet d6f253aa2cc740ba9d837af8fcebc0a6
I20260926 03:43:53.535413 2322 tablet_service.cc:1514] Processing DeleteTablet for tablet d6f253aa2cc740ba9d837af8fcebc0a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:43:53 UTC) from {username='kudu', principal='kudu/127.0.194.254@KRBTEST.COM'} at 127.0.0.1:36772
I20260926 03:43:53.535526 2044 tablet_service.cc:1514] Processing DeleteTablet for tablet d6f253aa2cc740ba9d837af8fcebc0a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:43:53 UTC) from {username='kudu', principal='kudu/127.0.194.254@KRBTEST.COM'} at 127.0.0.1:36392
I20260926 03:43:53.535580 2183 tablet_service.cc:1514] Processing DeleteTablet for tablet d6f253aa2cc740ba9d837af8fcebc0a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:43:53 UTC) from {username='kudu', principal='kudu/127.0.194.254@KRBTEST.COM'} at 127.0.0.1:44226
I20260926 03:43:53.537560 1905 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53016:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 03:43:53.538347 1905 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 03:43:53.548485 2043 tablet_service.cc:1467] Processing CreateTablet for tablet af4366574be04591b6ada84acdf05de3 (DEFAULT_TABLE table=test-table [id=ea0d984049ef42dcabb201d6e2d69461]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:43:53.548856 2182 tablet_service.cc:1467] Processing CreateTablet for tablet af4366574be04591b6ada84acdf05de3 (DEFAULT_TABLE table=test-table [id=ea0d984049ef42dcabb201d6e2d69461]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:43:53.549110 2043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af4366574be04591b6ada84acdf05de3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:43:53.549453 2182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af4366574be04591b6ada84acdf05de3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:43:53.551753 2321 tablet_service.cc:1467] Processing CreateTablet for tablet af4366574be04591b6ada84acdf05de3 (DEFAULT_TABLE table=test-table [id=ea0d984049ef42dcabb201d6e2d69461]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:43:53.552449 2321 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af4366574be04591b6ada84acdf05de3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:43:53.556072 2413 tablet_bootstrap.cc:492] T af4366574be04591b6ada84acdf05de3 P df904b94fdc14a05b7d78665d955f86f: Bootstrap starting.
I20260926 03:43:53.557731 2413 tablet_bootstrap.cc:654] T af4366574be04591b6ada84acdf05de3 P df904b94fdc14a05b7d78665d955f86f: Neither blocks nor log segments found. Creating new log.
I20260926 03:43:53.558915 2411 tablet_bootstrap.cc:492] T af4366574be04591b6ada84acdf05de3 P 88b981e00e3e43089b67f936a8e86e51: Bootstrap starting.
I20260926 03:43:53.560019 2434 tablet_replica.cc:330] T d6f253aa2cc740ba9d837af8fcebc0a6 P cdbcd498e18240d88a06242e77df5360: stopping tablet replica
I20260926 03:43:53.560499 2434 raft_consensus.cc:2229] T d6f253aa2cc740ba9d837af8fcebc0a6 P cdbcd498e18240d88a06242e77df5360 [term 1 LEADER]: Raft consensus shutting down.
I20260926 03:43:53.560642 2411 tablet_bootstrap.cc:654] T af4366574be04591b6ada84acdf05de3 P 88b981e00e3e43089b67f936a8e86e51: Neither blocks nor log segments found. Creating new log.
I20260926 03:43:53.561286 2412 tablet_bootstrap.cc:492] T af4366574be04591b6ada84acdf05de3 P cdbcd498e18240d88a06242e77df5360: Bootstrap starting.
I20260926 03:43:53.562307 2435 tablet_replica.cc:330] T d6f253aa2cc740ba9d837af8fcebc0a6 P 88b981e00e3e43089b67f936a8e86e51: stopping tablet replica
I20260926 03:43:53.562876 2435 raft_consensus.cc:2229] T d6f253aa2cc740ba9d837af8fcebc0a6 P 88b981e00e3e43089b67f936a8e86e51 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:43:53.562958 2412 tablet_bootstrap.cc:654] T af4366574be04591b6ada84acdf05de3 P cdbcd498e18240d88a06242e77df5360: Neither blocks nor log segments found. Creating new log.
I20260926 03:43:53.563664 2434 raft_consensus.cc:2258] T d6f253aa2cc740ba9d837af8fcebc0a6 P cdbcd498e18240d88a06242e77df5360 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:43:53.564677 2435 raft_consensus.cc:2258] T d6f253aa2cc740ba9d837af8fcebc0a6 P 88b981e00e3e43089b67f936a8e86e51 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:43:53.564832 2436 tablet_replica.cc:330] T d6f253aa2cc740ba9d837af8fcebc0a6 P df904b94fdc14a05b7d78665d955f86f: stopping tablet replica
I20260926 03:43:53.565407 2436 raft_consensus.cc:2229] T d6f253aa2cc740ba9d837af8fcebc0a6 P df904b94fdc14a05b7d78665d955f86f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:43:53.566042 2436 raft_consensus.cc:2258] T d6f253aa2cc740ba9d837af8fcebc0a6 P df904b94fdc14a05b7d78665d955f86f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:43:53.568151 2436 ts_tablet_manager.cc:1905] T d6f253aa2cc740ba9d837af8fcebc0a6 P df904b94fdc14a05b7d78665d955f86f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:43:53.568151 2434 ts_tablet_manager.cc:1905] T d6f253aa2cc740ba9d837af8fcebc0a6 P cdbcd498e18240d88a06242e77df5360: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:43:53.568883 2411 tablet_bootstrap.cc:492] T af4366574be04591b6ada84acdf05de3 P 88b981e00e3e43089b67f936a8e86e51: No bootstrap required, opened a new log
I20260926 03:43:53.569033 2411 ts_tablet_manager.cc:1397] T af4366574be04591b6ada84acdf05de3 P 88b981e00e3e43089b67f936a8e86e51: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.005s
I20260926 03:43:53.569425 2413 tablet_bootstrap.cc:492] T af4366574be04591b6ada84acdf05de3 P df904b94fdc14a05b7d78665d955f86f: No bootstrap required, opened a new log
I20260926 03:43:53.569572 2413 ts_tablet_manager.cc:1397] T af4366574be04591b6ada84acdf05de3 P df904b94fdc14a05b7d78665d955f86f: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.004s
I20260926 03:43:53.569844 2411 raft_consensus.cc:348] T af4366574be04591b6ada84acdf05de3 P 88b981e00e3e43089b67f936a8e86e51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df904b94fdc14a05b7d78665d955f86f" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 33137 } } peers { permanent_uuid: "88b981e00e3e43089b67f936a8e86e51" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 46757 } } peers { permanent_uuid: "cdbcd498e18240d88a06242e77df5360" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 33291 } }
I20260926 03:43:53.570065 2411 raft_consensus.cc:374] T af4366574be04591b6ada84acdf05de3 P 88b981e00e3e43089b67f936a8e86e51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:43:53.570166 2411 raft_consensus.cc:729] T af4366574be04591b6ada84acdf05de3 P 88b981e00e3e43089b67f936a8e86e51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88b981e00e3e43089b67f936a8e86e51, State: Initialized, Role: FOLLOWER
I20260926 03:43:53.570462 2411 consensus_queue.cc:260] T af4366574be04591b6ada84acdf05de3 P 88b981e00e3e43089b67f936a8e86e51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df904b94fdc14a05b7d78665d955f86f" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 33137 } } peers { permanent_uuid: "88b981e00e3e43089b67f936a8e86e51" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 46757 } } peers { permanent_uuid: "cdbcd498e18240d88a06242e77df5360" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 33291 } }
I20260926 03:43:53.571205 2413 raft_consensus.cc:348] T af4366574be04591b6ada84acdf05de3 P df904b94fdc14a05b7d78665d955f86f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df904b94fdc14a05b7d78665d955f86f" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 33137 } } peers { permanent_uuid: "88b981e00e3e43089b67f936a8e86e51" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 46757 } } peers { permanent_uuid: "cdbcd498e18240d88a06242e77df5360" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 33291 } }
I20260926 03:43:53.571427 2413 raft_consensus.cc:374] T af4366574be04591b6ada84acdf05de3 P df904b94fdc14a05b7d78665d955f86f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:43:53.571540 2413 raft_consensus.cc:729] T af4366574be04591b6ada84acdf05de3 P df904b94fdc14a05b7d78665d955f86f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df904b94fdc14a05b7d78665d955f86f, State: Initialized, Role: FOLLOWER
I20260926 03:43:53.571853 2413 consensus_queue.cc:260] T af4366574be04591b6ada84acdf05de3 P df904b94fdc14a05b7d78665d955f86f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df904b94fdc14a05b7d78665d955f86f" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 33137 } } peers { permanent_uuid: "88b981e00e3e43089b67f936a8e86e51" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 46757 } } peers { permanent_uuid: "cdbcd498e18240d88a06242e77df5360" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 33291 } }
I20260926 03:43:53.572499 2412 tablet_bootstrap.cc:492] T af4366574be04591b6ada84acdf05de3 P cdbcd498e18240d88a06242e77df5360: No bootstrap required, opened a new log
I20260926 03:43:53.572644 2412 ts_tablet_manager.cc:1397] T af4366574be04591b6ada84acdf05de3 P cdbcd498e18240d88a06242e77df5360: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.005s
I20260926 03:43:53.573523 2412 raft_consensus.cc:348] T af4366574be04591b6ada84acdf05de3 P cdbcd498e18240d88a06242e77df5360 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df904b94fdc14a05b7d78665d955f86f" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 33137 } } peers { permanent_uuid: "88b981e00e3e43089b67f936a8e86e51" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 46757 } } peers { permanent_uuid: "cdbcd498e18240d88a06242e77df5360" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 33291 } }
I20260926 03:43:53.573822 2412 raft_consensus.cc:374] T af4366574be04591b6ada84acdf05de3 P cdbcd498e18240d88a06242e77df5360 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:43:53.573925 2412 raft_consensus.cc:729] T af4366574be04591b6ada84acdf05de3 P cdbcd498e18240d88a06242e77df5360 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cdbcd498e18240d88a06242e77df5360, State: Initialized, Role: FOLLOWER
I20260926 03:43:53.574177 2412 consensus_queue.cc:260] T af4366574be04591b6ada84acdf05de3 P cdbcd498e18240d88a06242e77df5360 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df904b94fdc14a05b7d78665d955f86f" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 33137 } } peers { permanent_uuid: "88b981e00e3e43089b67f936a8e86e51" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 46757 } } peers { permanent_uuid: "cdbcd498e18240d88a06242e77df5360" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 33291 } }
I20260926 03:43:53.574793 2411 ts_tablet_manager.cc:1428] T af4366574be04591b6ada84acdf05de3 P 88b981e00e3e43089b67f936a8e86e51: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20260926 03:43:53.573303 2413 ts_tablet_manager.cc:1428] T af4366574be04591b6ada84acdf05de3 P df904b94fdc14a05b7d78665d955f86f: Time spent starting tablet: real 0.004s user 0.002s sys 0.001s
I20260926 03:43:53.576330 2436 ts_tablet_manager.cc:1918] T d6f253aa2cc740ba9d837af8fcebc0a6 P df904b94fdc14a05b7d78665d955f86f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:43:53.576573 2436 log.cc:1196] T d6f253aa2cc740ba9d837af8fcebc0a6 P df904b94fdc14a05b7d78665d955f86f: Deleting WAL directory at /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790394221564982-779-0/minicluster-data/ts-0/wal/wals/d6f253aa2cc740ba9d837af8fcebc0a6
I20260926 03:43:53.577205 2436 ts_tablet_manager.cc:1939] T d6f253aa2cc740ba9d837af8fcebc0a6 P df904b94fdc14a05b7d78665d955f86f: Deleting consensus metadata
I20260926 03:43:53.577314 2412 ts_tablet_manager.cc:1428] T af4366574be04591b6ada84acdf05de3 P cdbcd498e18240d88a06242e77df5360: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260926 03:43:53.579170 1893 catalog_manager.cc:4841] TS df904b94fdc14a05b7d78665d955f86f (127.0.194.193:33137): tablet d6f253aa2cc740ba9d837af8fcebc0a6 (table test-table [id=43645a3081c74664b982f75f346c5f09]) successfully deleted
I20260926 03:43:53.580111 2435 ts_tablet_manager.cc:1905] T d6f253aa2cc740ba9d837af8fcebc0a6 P 88b981e00e3e43089b67f936a8e86e51: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:43:53.583058 2434 ts_tablet_manager.cc:1918] T d6f253aa2cc740ba9d837af8fcebc0a6 P cdbcd498e18240d88a06242e77df5360: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:43:53.583228 2434 log.cc:1196] T d6f253aa2cc740ba9d837af8fcebc0a6 P cdbcd498e18240d88a06242e77df5360: Deleting WAL directory at /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790394221564982-779-0/minicluster-data/ts-2/wal/wals/d6f253aa2cc740ba9d837af8fcebc0a6
I20260926 03:43:53.583746 2434 ts_tablet_manager.cc:1939] T d6f253aa2cc740ba9d837af8fcebc0a6 P cdbcd498e18240d88a06242e77df5360: Deleting consensus metadata
I20260926 03:43:53.585613 1891 catalog_manager.cc:4841] TS cdbcd498e18240d88a06242e77df5360 (127.0.194.195:33291): tablet d6f253aa2cc740ba9d837af8fcebc0a6 (table test-table [id=43645a3081c74664b982f75f346c5f09]) successfully deleted
I20260926 03:43:53.585966 2435 ts_tablet_manager.cc:1918] T d6f253aa2cc740ba9d837af8fcebc0a6 P 88b981e00e3e43089b67f936a8e86e51: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:43:53.586131 2435 log.cc:1196] T d6f253aa2cc740ba9d837af8fcebc0a6 P 88b981e00e3e43089b67f936a8e86e51: Deleting WAL directory at /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790394221564982-779-0/minicluster-data/ts-1/wal/wals/d6f253aa2cc740ba9d837af8fcebc0a6
I20260926 03:43:53.586709 2435 ts_tablet_manager.cc:1939] T d6f253aa2cc740ba9d837af8fcebc0a6 P 88b981e00e3e43089b67f936a8e86e51: Deleting consensus metadata
I20260926 03:43:53.588375 1892 catalog_manager.cc:4841] TS 88b981e00e3e43089b67f936a8e86e51 (127.0.194.194:46757): tablet d6f253aa2cc740ba9d837af8fcebc0a6 (table test-table [id=43645a3081c74664b982f75f346c5f09]) successfully deleted
I20260926 03:43:53.678076 2417 raft_consensus.cc:482] T af4366574be04591b6ada84acdf05de3 P 88b981e00e3e43089b67f936a8e86e51 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 03:43:53.678350 2417 raft_consensus.cc:504] T af4366574be04591b6ada84acdf05de3 P 88b981e00e3e43089b67f936a8e86e51 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df904b94fdc14a05b7d78665d955f86f" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 33137 } } peers { permanent_uuid: "88b981e00e3e43089b67f936a8e86e51" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 46757 } } peers { permanent_uuid: "cdbcd498e18240d88a06242e77df5360" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 33291 } }
I20260926 03:43:53.680060 2417 leader_election.cc:288] T af4366574be04591b6ada84acdf05de3 P 88b981e00e3e43089b67f936a8e86e51 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df904b94fdc14a05b7d78665d955f86f (127.0.194.193:33137), cdbcd498e18240d88a06242e77df5360 (127.0.194.195:33291)
I20260926 03:43:53.689308 2406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:53.680610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.194:50807 (local address 127.0.194.195:33291)
0926 03:43:53.680847 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0926 03:43:53.680857 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 03:43:53.682329 (+ 1472us) server_negotiation.cc:407] Connection header received
0926 03:43:53.682499 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:43:53.682508 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:43:53.682589 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:43:53.682688 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:43:53.683455 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:53.684649 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:53.687401 (+ 2752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:53.688277 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:53.688372 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:43:53.688427 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 03:43:53.688882 (+ 455us) server_negotiation.cc:299] Negotiation successful
0926 03:43:53.689006 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260926 03:43:53.689396 2441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:53.680420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.195:33291 (local address 127.0.194.194:50807)
0926 03:43:53.682231 (+ 1811us) negotiation.cc:107] Waiting for socket to connect
0926 03:43:53.682245 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 03:43:53.682364 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:43:53.682752 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:43:53.682761 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:43:53.682788 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:43:53.683302 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:43:53.683316 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:43:53.684843 (+ 1527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:43:53.684854 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:43:53.687209 (+ 2355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:43:53.687223 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:43:53.688587 (+ 1364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:43:53.688592 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:43:53.688719 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:43:53.688742 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 03:43:53.688846 (+ 104us) client_negotiation.cc:240] Negotiation successful
0926 03:43:53.688985 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1681,"mutex_wait_us":25,"thread_start_us":1440,"threads_started":1}
I20260926 03:43:53.689800 2409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:53.681142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.194:40621 (local address 127.0.194.193:33137)
0926 03:43:53.681361 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0926 03:43:53.681369 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:43:53.681968 (+ 599us) server_negotiation.cc:407] Connection header received
0926 03:43:53.682269 (+ 301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:43:53.682275 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:43:53.682355 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:43:53.682452 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:43:53.683587 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:53.684659 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:53.687331 (+ 2672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:53.688596 (+ 1265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:53.688730 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:43:53.688801 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 03:43:53.689399 (+ 598us) server_negotiation.cc:299] Negotiation successful
0926 03:43:53.689537 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260926 03:43:53.690234 2345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af4366574be04591b6ada84acdf05de3" candidate_uuid: "88b981e00e3e43089b67f936a8e86e51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdbcd498e18240d88a06242e77df5360" is_pre_election: true
I20260926 03:43:53.690501 2440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:53.680300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.193:33137 (local address 127.0.194.194:40621)
0926 03:43:53.681837 (+ 1537us) negotiation.cc:107] Waiting for socket to connect
0926 03:43:53.681863 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0926 03:43:53.682087 (+ 224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:43:53.682834 (+ 747us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:43:53.682839 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:43:53.682877 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:43:53.683330 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:43:53.683345 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:43:53.684789 (+ 1444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:43:53.684796 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:43:53.687151 (+ 2355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:43:53.687166 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:43:53.688991 (+ 1825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:43:53.688997 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:43:53.689202 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:43:53.689236 (+ 34us) client_negotiation.cc:769] Sending connection context
0926 03:43:53.689974 (+ 738us) client_negotiation.cc:240] Negotiation successful
0926 03:43:53.690124 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1389,"mutex_wait_us":32,"thread_start_us":1238,"threads_started":1}
I20260926 03:43:53.690676 2345 raft_consensus.cc:2454] T af4366574be04591b6ada84acdf05de3 P cdbcd498e18240d88a06242e77df5360 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88b981e00e3e43089b67f936a8e86e51 in term 0.
I20260926 03:43:53.691177 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af4366574be04591b6ada84acdf05de3" candidate_uuid: "88b981e00e3e43089b67f936a8e86e51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df904b94fdc14a05b7d78665d955f86f" is_pre_election: true
I20260926 03:43:53.691275 2134 leader_election.cc:302] T af4366574be04591b6ada84acdf05de3 P 88b981e00e3e43089b67f936a8e86e51 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88b981e00e3e43089b67f936a8e86e51, cdbcd498e18240d88a06242e77df5360; no voters:
I20260926 03:43:53.691412 2064 raft_consensus.cc:2454] T af4366574be04591b6ada84acdf05de3 P df904b94fdc14a05b7d78665d955f86f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88b981e00e3e43089b67f936a8e86e51 in term 0.
I20260926 03:43:53.691720 2417 raft_consensus.cc:2781] T af4366574be04591b6ada84acdf05de3 P 88b981e00e3e43089b67f936a8e86e51 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 03:43:53.691849 2417 raft_consensus.cc:482] T af4366574be04591b6ada84acdf05de3 P 88b981e00e3e43089b67f936a8e86e51 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 03:43:53.691963 2417 raft_consensus.cc:3037] T af4366574be04591b6ada84acdf05de3 P 88b981e00e3e43089b67f936a8e86e51 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:43:53.694721 2417 raft_consensus.cc:504] T af4366574be04591b6ada84acdf05de3 P 88b981e00e3e43089b67f936a8e86e51 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df904b94fdc14a05b7d78665d955f86f" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 33137 } } peers { permanent_uuid: "88b981e00e3e43089b67f936a8e86e51" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 46757 } } peers { permanent_uuid: "cdbcd498e18240d88a06242e77df5360" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 33291 } }
I20260926 03:43:53.695907 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af4366574be04591b6ada84acdf05de3" candidate_uuid: "88b981e00e3e43089b67f936a8e86e51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df904b94fdc14a05b7d78665d955f86f"
I20260926 03:43:53.695955 2345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af4366574be04591b6ada84acdf05de3" candidate_uuid: "88b981e00e3e43089b67f936a8e86e51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdbcd498e18240d88a06242e77df5360"
I20260926 03:43:53.696120 2064 raft_consensus.cc:3037] T af4366574be04591b6ada84acdf05de3 P df904b94fdc14a05b7d78665d955f86f [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:43:53.696162 2345 raft_consensus.cc:3037] T af4366574be04591b6ada84acdf05de3 P cdbcd498e18240d88a06242e77df5360 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:43:53.698449 2345 raft_consensus.cc:2454] T af4366574be04591b6ada84acdf05de3 P cdbcd498e18240d88a06242e77df5360 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88b981e00e3e43089b67f936a8e86e51 in term 1.
I20260926 03:43:53.698706 2417 leader_election.cc:288] T af4366574be04591b6ada84acdf05de3 P 88b981e00e3e43089b67f936a8e86e51 [CANDIDATE]: Term 1 election: Requested vote from peers df904b94fdc14a05b7d78665d955f86f (127.0.194.193:33137), cdbcd498e18240d88a06242e77df5360 (127.0.194.195:33291)
I20260926 03:43:53.698925 2134 leader_election.cc:302] T af4366574be04591b6ada84acdf05de3 P 88b981e00e3e43089b67f936a8e86e51 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88b981e00e3e43089b67f936a8e86e51, cdbcd498e18240d88a06242e77df5360; no voters:
I20260926 03:43:53.699142 2064 raft_consensus.cc:2454] T af4366574be04591b6ada84acdf05de3 P df904b94fdc14a05b7d78665d955f86f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88b981e00e3e43089b67f936a8e86e51 in term 1.
I20260926 03:43:53.699234 2417 raft_consensus.cc:2781] T af4366574be04591b6ada84acdf05de3 P 88b981e00e3e43089b67f936a8e86e51 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:43:53.699747 2417 raft_consensus.cc:686] T af4366574be04591b6ada84acdf05de3 P 88b981e00e3e43089b67f936a8e86e51 [term 1 LEADER]: Becoming Leader. State: Replica: 88b981e00e3e43089b67f936a8e86e51, State: Running, Role: LEADER
I20260926 03:43:53.700168 2417 consensus_queue.cc:237] T af4366574be04591b6ada84acdf05de3 P 88b981e00e3e43089b67f936a8e86e51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df904b94fdc14a05b7d78665d955f86f" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 33137 } } peers { permanent_uuid: "88b981e00e3e43089b67f936a8e86e51" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 46757 } } peers { permanent_uuid: "cdbcd498e18240d88a06242e77df5360" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 33291 } }
I20260926 03:43:53.703773 1906 catalog_manager.cc:5495] T af4366574be04591b6ada84acdf05de3 P 88b981e00e3e43089b67f936a8e86e51 reported cstate change: term changed from 0 to 1, leader changed from <none> to 88b981e00e3e43089b67f936a8e86e51 (127.0.194.194). New cstate: current_term: 1 leader_uuid: "88b981e00e3e43089b67f936a8e86e51" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df904b94fdc14a05b7d78665d955f86f" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 33137 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88b981e00e3e43089b67f936a8e86e51" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 46757 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cdbcd498e18240d88a06242e77df5360" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 33291 } health_report { overall_health: UNKNOWN } } }
Sep 26 03:43:53 dist-test-slave-n2th krb5kdc[1862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394231, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.254@KRBTEST.COM for kudu/127.0.194.254@KRBTEST.COM
I20260926 03:43:53.775123 2449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:53.761335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:37875 (local address 127.0.0.1:53024)
0926 03:43:53.762011 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0926 03:43:53.762028 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 03:43:53.762198 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:43:53.762603 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:43:53.762606 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:43:53.763434 (+ 828us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:43:53.763685 (+ 251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:43:53.763694 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:43:53.764678 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:43:53.764685 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:43:53.765666 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:43:53.765678 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:43:53.766412 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:43:53.766418 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:43:53.766575 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:43:53.767344 (+ 769us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:43:53.767361 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:43:53.769664 (+ 2303us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:43:53.772546 (+ 2882us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:43:53.772559 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:43:53.772566 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:43:53.772975 (+ 409us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:43:53.773338 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:43:53.773345 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:43:53.773349 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:43:53.773472 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:43:53.773977 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:43:53.773985 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:43:53.774169 (+ 184us) client_negotiation.cc:769] Sending connection context
0926 03:43:53.774332 (+ 163us) client_negotiation.cc:240] Negotiation successful
0926 03:43:53.774634 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":540,"thread_start_us":343,"threads_started":1}
I20260926 03:43:53.775143 2002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:53.761244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53024 (local address 127.0.194.254:37875)
0926 03:43:53.761489 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0926 03:43:53.761495 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:43:53.762120 (+ 625us) server_negotiation.cc:407] Connection header received
0926 03:43:53.762366 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:43:53.762371 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:43:53.762462 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:43:53.762556 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:43:53.763830 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:53.764546 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:53.765806 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:53.766237 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:53.766377 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:43:53.769831 (+ 3454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:43:53.769841 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:43:53.769845 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:43:53.769871 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:43:53.772409 (+ 2538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:43:53.773092 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:43:53.773100 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:43:53.773104 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:43:53.773233 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:43:53.773572 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:43:53.773579 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:43:53.773582 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:43:53.773831 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:43:53.774015 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0926 03:43:53.774428 (+ 413us) server_negotiation.cc:299] Negotiation successful
0926 03:43:53.774805 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":138,"server-negotiator.queue_time_us":56}
I20260926 03:43:53.780644 1906 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.194.254@KRBTEST.COM'} at 127.0.0.1:53024:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260926 03:43:53.781929 1906 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 03:43:53.788960 2044 tablet_service.cc:1467] Processing CreateTablet for tablet fffd1e2b87f2458eba56a1d319da8706 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=cd01619435f347e787ea2368d45a05a3]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 03:43:53.789134 2321 tablet_service.cc:1467] Processing CreateTablet for tablet fffd1e2b87f2458eba56a1d319da8706 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=cd01619435f347e787ea2368d45a05a3]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 03:43:53.789573 2044 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fffd1e2b87f2458eba56a1d319da8706. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:43:53.789645 2321 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fffd1e2b87f2458eba56a1d319da8706. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:43:53.794710 2412 tablet_bootstrap.cc:492] T fffd1e2b87f2458eba56a1d319da8706 P cdbcd498e18240d88a06242e77df5360: Bootstrap starting.
I20260926 03:43:53.795509 2183 tablet_service.cc:1467] Processing CreateTablet for tablet fffd1e2b87f2458eba56a1d319da8706 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=cd01619435f347e787ea2368d45a05a3]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 03:43:53.796013 2183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fffd1e2b87f2458eba56a1d319da8706. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:43:53.796186 2412 tablet_bootstrap.cc:654] T fffd1e2b87f2458eba56a1d319da8706 P cdbcd498e18240d88a06242e77df5360: Neither blocks nor log segments found. Creating new log.
I20260926 03:43:53.798386 2412 tablet_bootstrap.cc:492] T fffd1e2b87f2458eba56a1d319da8706 P cdbcd498e18240d88a06242e77df5360: No bootstrap required, opened a new log
I20260926 03:43:53.798575 2412 ts_tablet_manager.cc:1397] T fffd1e2b87f2458eba56a1d319da8706 P cdbcd498e18240d88a06242e77df5360: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20260926 03:43:53.799340 2412 raft_consensus.cc:348] T fffd1e2b87f2458eba56a1d319da8706 P cdbcd498e18240d88a06242e77df5360 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df904b94fdc14a05b7d78665d955f86f" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 33137 } } peers { permanent_uuid: "88b981e00e3e43089b67f936a8e86e51" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 46757 } } peers { permanent_uuid: "cdbcd498e18240d88a06242e77df5360" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 33291 } }
I20260926 03:43:53.799604 2412 raft_consensus.cc:374] T fffd1e2b87f2458eba56a1d319da8706 P cdbcd498e18240d88a06242e77df5360 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:43:53.799693 2412 raft_consensus.cc:729] T fffd1e2b87f2458eba56a1d319da8706 P cdbcd498e18240d88a06242e77df5360 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cdbcd498e18240d88a06242e77df5360, State: Initialized, Role: FOLLOWER
I20260926 03:43:53.799939 2412 consensus_queue.cc:260] T fffd1e2b87f2458eba56a1d319da8706 P cdbcd498e18240d88a06242e77df5360 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df904b94fdc14a05b7d78665d955f86f" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 33137 } } peers { permanent_uuid: "88b981e00e3e43089b67f936a8e86e51" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 46757 } } peers { permanent_uuid: "cdbcd498e18240d88a06242e77df5360" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 33291 } }
I20260926 03:43:53.800911 2431 tablet_replica.cc:439] T fffd1e2b87f2458eba56a1d319da8706 P cdbcd498e18240d88a06242e77df5360: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df904b94fdc14a05b7d78665d955f86f" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 33137 } } peers { permanent_uuid: "88b981e00e3e43089b67f936a8e86e51" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 46757 } } peers { permanent_uuid: "cdbcd498e18240d88a06242e77df5360" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 33291 } } }
I20260926 03:43:53.801316 2431 tablet_replica.cc:442] T fffd1e2b87f2458eba56a1d319da8706 P cdbcd498e18240d88a06242e77df5360: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 03:43:53.802428 2412 ts_tablet_manager.cc:1428] T fffd1e2b87f2458eba56a1d319da8706 P cdbcd498e18240d88a06242e77df5360: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260926 03:43:53.803102 2411 tablet_bootstrap.cc:492] T fffd1e2b87f2458eba56a1d319da8706 P 88b981e00e3e43089b67f936a8e86e51: Bootstrap starting.
I20260926 03:43:53.804811 2411 tablet_bootstrap.cc:654] T fffd1e2b87f2458eba56a1d319da8706 P 88b981e00e3e43089b67f936a8e86e51: Neither blocks nor log segments found. Creating new log.
I20260926 03:43:53.807356 2411 tablet_bootstrap.cc:492] T fffd1e2b87f2458eba56a1d319da8706 P 88b981e00e3e43089b67f936a8e86e51: No bootstrap required, opened a new log
I20260926 03:43:53.807529 2411 ts_tablet_manager.cc:1397] T fffd1e2b87f2458eba56a1d319da8706 P 88b981e00e3e43089b67f936a8e86e51: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20260926 03:43:53.807690 2413 tablet_bootstrap.cc:492] T fffd1e2b87f2458eba56a1d319da8706 P df904b94fdc14a05b7d78665d955f86f: Bootstrap starting.
I20260926 03:43:53.808269 2411 raft_consensus.cc:348] T fffd1e2b87f2458eba56a1d319da8706 P 88b981e00e3e43089b67f936a8e86e51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df904b94fdc14a05b7d78665d955f86f" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 33137 } } peers { permanent_uuid: "88b981e00e3e43089b67f936a8e86e51" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 46757 } } peers { permanent_uuid: "cdbcd498e18240d88a06242e77df5360" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 33291 } }
I20260926 03:43:53.808537 2411 raft_consensus.cc:374] T fffd1e2b87f2458eba56a1d319da8706 P 88b981e00e3e43089b67f936a8e86e51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:43:53.808694 2411 raft_consensus.cc:729] T fffd1e2b87f2458eba56a1d319da8706 P 88b981e00e3e43089b67f936a8e86e51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88b981e00e3e43089b67f936a8e86e51, State: Initialized, Role: FOLLOWER
I20260926 03:43:53.808986 2411 consensus_queue.cc:260] T fffd1e2b87f2458eba56a1d319da8706 P 88b981e00e3e43089b67f936a8e86e51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df904b94fdc14a05b7d78665d955f86f" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 33137 } } peers { permanent_uuid: "88b981e00e3e43089b67f936a8e86e51" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 46757 } } peers { permanent_uuid: "cdbcd498e18240d88a06242e77df5360" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 33291 } }
I20260926 03:43:53.809281 2413 tablet_bootstrap.cc:654] T fffd1e2b87f2458eba56a1d319da8706 P df904b94fdc14a05b7d78665d955f86f: Neither blocks nor log segments found. Creating new log.
I20260926 03:43:53.809414 2417 tablet_replica.cc:439] T fffd1e2b87f2458eba56a1d319da8706 P 88b981e00e3e43089b67f936a8e86e51: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df904b94fdc14a05b7d78665d955f86f" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 33137 } } peers { permanent_uuid: "88b981e00e3e43089b67f936a8e86e51" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 46757 } } peers { permanent_uuid: "cdbcd498e18240d88a06242e77df5360" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 33291 } } }
I20260926 03:43:53.809679 2417 tablet_replica.cc:442] T fffd1e2b87f2458eba56a1d319da8706 P 88b981e00e3e43089b67f936a8e86e51: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 03:43:53.811020 2411 ts_tablet_manager.cc:1428] T fffd1e2b87f2458eba56a1d319da8706 P 88b981e00e3e43089b67f936a8e86e51: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260926 03:43:53.811860 2413 tablet_bootstrap.cc:492] T fffd1e2b87f2458eba56a1d319da8706 P df904b94fdc14a05b7d78665d955f86f: No bootstrap required, opened a new log
I20260926 03:43:53.812019 2413 ts_tablet_manager.cc:1397] T fffd1e2b87f2458eba56a1d319da8706 P df904b94fdc14a05b7d78665d955f86f: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s
I20260926 03:43:53.812738 2413 raft_consensus.cc:348] T fffd1e2b87f2458eba56a1d319da8706 P df904b94fdc14a05b7d78665d955f86f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df904b94fdc14a05b7d78665d955f86f" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 33137 } } peers { permanent_uuid: "88b981e00e3e43089b67f936a8e86e51" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 46757 } } peers { permanent_uuid: "cdbcd498e18240d88a06242e77df5360" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 33291 } }
I20260926 03:43:53.813087 2413 raft_consensus.cc:374] T fffd1e2b87f2458eba56a1d319da8706 P df904b94fdc14a05b7d78665d955f86f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:43:53.813184 2413 raft_consensus.cc:729] T fffd1e2b87f2458eba56a1d319da8706 P df904b94fdc14a05b7d78665d955f86f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df904b94fdc14a05b7d78665d955f86f, State: Initialized, Role: FOLLOWER
I20260926 03:43:53.813433 2413 consensus_queue.cc:260] T fffd1e2b87f2458eba56a1d319da8706 P df904b94fdc14a05b7d78665d955f86f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df904b94fdc14a05b7d78665d955f86f" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 33137 } } peers { permanent_uuid: "88b981e00e3e43089b67f936a8e86e51" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 46757 } } peers { permanent_uuid: "cdbcd498e18240d88a06242e77df5360" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 33291 } }
I20260926 03:43:53.814100 2413 ts_tablet_manager.cc:1428] T fffd1e2b87f2458eba56a1d319da8706 P df904b94fdc14a05b7d78665d955f86f: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260926 03:43:53.813879 2419 tablet_replica.cc:439] T fffd1e2b87f2458eba56a1d319da8706 P df904b94fdc14a05b7d78665d955f86f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df904b94fdc14a05b7d78665d955f86f" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 33137 } } peers { permanent_uuid: "88b981e00e3e43089b67f936a8e86e51" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 46757 } } peers { permanent_uuid: "cdbcd498e18240d88a06242e77df5360" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 33291 } } }
I20260926 03:43:53.814236 2419 tablet_replica.cc:442] T fffd1e2b87f2458eba56a1d319da8706 P df904b94fdc14a05b7d78665d955f86f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 03:43:53.913914 2417 raft_consensus.cc:482] T fffd1e2b87f2458eba56a1d319da8706 P 88b981e00e3e43089b67f936a8e86e51 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 03:43:53.914199 2417 raft_consensus.cc:504] T fffd1e2b87f2458eba56a1d319da8706 P 88b981e00e3e43089b67f936a8e86e51 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df904b94fdc14a05b7d78665d955f86f" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 33137 } } peers { permanent_uuid: "88b981e00e3e43089b67f936a8e86e51" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 46757 } } peers { permanent_uuid: "cdbcd498e18240d88a06242e77df5360" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 33291 } }
I20260926 03:43:53.914937 2417 leader_election.cc:288] T fffd1e2b87f2458eba56a1d319da8706 P 88b981e00e3e43089b67f936a8e86e51 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df904b94fdc14a05b7d78665d955f86f (127.0.194.193:33137), cdbcd498e18240d88a06242e77df5360 (127.0.194.195:33291)
I20260926 03:43:53.915529 2345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fffd1e2b87f2458eba56a1d319da8706" candidate_uuid: "88b981e00e3e43089b67f936a8e86e51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdbcd498e18240d88a06242e77df5360" is_pre_election: true
I20260926 03:43:53.915524 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fffd1e2b87f2458eba56a1d319da8706" candidate_uuid: "88b981e00e3e43089b67f936a8e86e51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df904b94fdc14a05b7d78665d955f86f" is_pre_election: true
I20260926 03:43:53.915804 2064 raft_consensus.cc:2454] T fffd1e2b87f2458eba56a1d319da8706 P df904b94fdc14a05b7d78665d955f86f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88b981e00e3e43089b67f936a8e86e51 in term 0.
I20260926 03:43:53.915820 2345 raft_consensus.cc:2454] T fffd1e2b87f2458eba56a1d319da8706 P cdbcd498e18240d88a06242e77df5360 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88b981e00e3e43089b67f936a8e86e51 in term 0.
I20260926 03:43:53.916289 2136 leader_election.cc:302] T fffd1e2b87f2458eba56a1d319da8706 P 88b981e00e3e43089b67f936a8e86e51 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88b981e00e3e43089b67f936a8e86e51, df904b94fdc14a05b7d78665d955f86f; no voters:
I20260926 03:43:53.916610 2417 raft_consensus.cc:2781] T fffd1e2b87f2458eba56a1d319da8706 P 88b981e00e3e43089b67f936a8e86e51 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 03:43:53.916723 2417 raft_consensus.cc:482] T fffd1e2b87f2458eba56a1d319da8706 P 88b981e00e3e43089b67f936a8e86e51 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 03:43:53.916851 2417 raft_consensus.cc:3037] T fffd1e2b87f2458eba56a1d319da8706 P 88b981e00e3e43089b67f936a8e86e51 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:43:53.920038 2417 raft_consensus.cc:504] T fffd1e2b87f2458eba56a1d319da8706 P 88b981e00e3e43089b67f936a8e86e51 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df904b94fdc14a05b7d78665d955f86f" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 33137 } } peers { permanent_uuid: "88b981e00e3e43089b67f936a8e86e51" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 46757 } } peers { permanent_uuid: "cdbcd498e18240d88a06242e77df5360" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 33291 } }
I20260926 03:43:53.921151 2417 leader_election.cc:288] T fffd1e2b87f2458eba56a1d319da8706 P 88b981e00e3e43089b67f936a8e86e51 [CANDIDATE]: Term 1 election: Requested vote from peers df904b94fdc14a05b7d78665d955f86f (127.0.194.193:33137), cdbcd498e18240d88a06242e77df5360 (127.0.194.195:33291)
I20260926 03:43:53.921289 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fffd1e2b87f2458eba56a1d319da8706" candidate_uuid: "88b981e00e3e43089b67f936a8e86e51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df904b94fdc14a05b7d78665d955f86f"
I20260926 03:43:53.921545 2064 raft_consensus.cc:3037] T fffd1e2b87f2458eba56a1d319da8706 P df904b94fdc14a05b7d78665d955f86f [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:43:53.921919 2345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fffd1e2b87f2458eba56a1d319da8706" candidate_uuid: "88b981e00e3e43089b67f936a8e86e51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdbcd498e18240d88a06242e77df5360"
I20260926 03:43:53.922159 2345 raft_consensus.cc:3037] T fffd1e2b87f2458eba56a1d319da8706 P cdbcd498e18240d88a06242e77df5360 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:43:53.923882 2064 raft_consensus.cc:2454] T fffd1e2b87f2458eba56a1d319da8706 P df904b94fdc14a05b7d78665d955f86f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88b981e00e3e43089b67f936a8e86e51 in term 1.
I20260926 03:43:53.924357 2136 leader_election.cc:302] T fffd1e2b87f2458eba56a1d319da8706 P 88b981e00e3e43089b67f936a8e86e51 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88b981e00e3e43089b67f936a8e86e51, df904b94fdc14a05b7d78665d955f86f; no voters:
I20260926 03:43:53.924633 2417 raft_consensus.cc:2781] T fffd1e2b87f2458eba56a1d319da8706 P 88b981e00e3e43089b67f936a8e86e51 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:43:53.925007 2417 raft_consensus.cc:686] T fffd1e2b87f2458eba56a1d319da8706 P 88b981e00e3e43089b67f936a8e86e51 [term 1 LEADER]: Becoming Leader. State: Replica: 88b981e00e3e43089b67f936a8e86e51, State: Running, Role: LEADER
I20260926 03:43:53.925278 2417 consensus_queue.cc:237] T fffd1e2b87f2458eba56a1d319da8706 P 88b981e00e3e43089b67f936a8e86e51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df904b94fdc14a05b7d78665d955f86f" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 33137 } } peers { permanent_uuid: "88b981e00e3e43089b67f936a8e86e51" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 46757 } } peers { permanent_uuid: "cdbcd498e18240d88a06242e77df5360" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 33291 } }
I20260926 03:43:53.925411 2345 raft_consensus.cc:2454] T fffd1e2b87f2458eba56a1d319da8706 P cdbcd498e18240d88a06242e77df5360 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88b981e00e3e43089b67f936a8e86e51 in term 1.
I20260926 03:43:53.927286 2442 tablet_replica.cc:439] T fffd1e2b87f2458eba56a1d319da8706 P 88b981e00e3e43089b67f936a8e86e51: TxnStatusTablet state changed. Reason: New leader 88b981e00e3e43089b67f936a8e86e51. Latest consensus state: current_term: 1 leader_uuid: "88b981e00e3e43089b67f936a8e86e51" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df904b94fdc14a05b7d78665d955f86f" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 33137 } } peers { permanent_uuid: "88b981e00e3e43089b67f936a8e86e51" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 46757 } } peers { permanent_uuid: "cdbcd498e18240d88a06242e77df5360" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 33291 } } }
I20260926 03:43:53.927706 2442 tablet_replica.cc:442] T fffd1e2b87f2458eba56a1d319da8706 P 88b981e00e3e43089b67f936a8e86e51: This TxnStatusTablet replica's current role is: LEADER
I20260926 03:43:53.928551 2454 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260926 03:43:53.928825 2454 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260926 03:43:53.929024 1905 catalog_manager.cc:5495] T fffd1e2b87f2458eba56a1d319da8706 P 88b981e00e3e43089b67f936a8e86e51 reported cstate change: term changed from 0 to 1, leader changed from <none> to 88b981e00e3e43089b67f936a8e86e51 (127.0.194.194). New cstate: current_term: 1 leader_uuid: "88b981e00e3e43089b67f936a8e86e51" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df904b94fdc14a05b7d78665d955f86f" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 33137 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88b981e00e3e43089b67f936a8e86e51" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 46757 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cdbcd498e18240d88a06242e77df5360" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 33291 } health_report { overall_health: UNKNOWN } } }
I20260926 03:43:53.968966 2449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:53.960963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.194:46757 (local address 127.0.0.1:44228)
0926 03:43:53.961175 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0926 03:43:53.961198 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 03:43:53.961379 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:43:53.962052 (+ 673us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:43:53.962058 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:43:53.962071 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 03:43:53.962299 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:43:53.962310 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:43:53.964143 (+ 1833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:43:53.964149 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:43:53.965803 (+ 1654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:43:53.965819 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:43:53.966962 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:43:53.966970 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:43:53.967178 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:43:53.967255 (+ 77us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 03:43:53.968371 (+ 1116us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 03:43:53.968394 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 03:43:53.968489 (+ 95us) client_negotiation.cc:240] Negotiation successful
0926 03:43:53.968649 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260926 03:43:53.968986 2457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:53.960530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44228 (local address 127.0.194.194:46757)
0926 03:43:53.961119 (+ 589us) server_negotiation.cc:206] Beginning negotiation
0926 03:43:53.961125 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:43:53.961281 (+ 156us) server_negotiation.cc:407] Connection header received
0926 03:43:53.961618 (+ 337us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:43:53.961625 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:43:53.961777 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:43:53.962469 (+ 692us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:43:53.962654 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:53.963935 (+ 1281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:53.965968 (+ 2033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:53.966687 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:53.966910 (+ 223us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:43:53.967455 (+ 545us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:43:53.968243 (+ 788us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:43:53.968397 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0926 03:43:53.968544 (+ 147us) server_negotiation.cc:299] Negotiation successful
0926 03:43:53.968636 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":438,"thread_start_us":286,"threads_started":1}
I20260926 03:43:53.973835 2064 raft_consensus.cc:1261] T fffd1e2b87f2458eba56a1d319da8706 P df904b94fdc14a05b7d78665d955f86f [term 1 FOLLOWER]: Refusing update from remote peer 88b981e00e3e43089b67f936a8e86e51: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:43:53.973901 2345 raft_consensus.cc:1261] T fffd1e2b87f2458eba56a1d319da8706 P cdbcd498e18240d88a06242e77df5360 [term 1 FOLLOWER]: Refusing update from remote peer 88b981e00e3e43089b67f936a8e86e51: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:43:53.974761 2442 consensus_queue.cc:1035] T fffd1e2b87f2458eba56a1d319da8706 P 88b981e00e3e43089b67f936a8e86e51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df904b94fdc14a05b7d78665d955f86f" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 33137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:43:53.975212 2417 consensus_queue.cc:1035] T fffd1e2b87f2458eba56a1d319da8706 P 88b981e00e3e43089b67f936a8e86e51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdbcd498e18240d88a06242e77df5360" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 33291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:43:53.984472 2419 tablet_replica.cc:439] T fffd1e2b87f2458eba56a1d319da8706 P df904b94fdc14a05b7d78665d955f86f: TxnStatusTablet state changed. Reason: New leader 88b981e00e3e43089b67f936a8e86e51. Latest consensus state: current_term: 1 leader_uuid: "88b981e00e3e43089b67f936a8e86e51" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df904b94fdc14a05b7d78665d955f86f" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 33137 } } peers { permanent_uuid: "88b981e00e3e43089b67f936a8e86e51" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 46757 } } peers { permanent_uuid: "cdbcd498e18240d88a06242e77df5360" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 33291 } } }
I20260926 03:43:53.984764 2419 tablet_replica.cc:442] T fffd1e2b87f2458eba56a1d319da8706 P df904b94fdc14a05b7d78665d955f86f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 03:43:53.990091 2425 mvcc.cc:205] Tried to move back new op lower bound from 7333454782348324864 to 7333454782161035264. Current Snapshot: MvccSnapshot[applied={T|T < 7333454782348324864 or (T in {7333454782348324864})}]
I20260926 03:43:53.990687 2419 tablet_replica.cc:439] T fffd1e2b87f2458eba56a1d319da8706 P df904b94fdc14a05b7d78665d955f86f: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "88b981e00e3e43089b67f936a8e86e51" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df904b94fdc14a05b7d78665d955f86f" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 33137 } } peers { permanent_uuid: "88b981e00e3e43089b67f936a8e86e51" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 46757 } } peers { permanent_uuid: "cdbcd498e18240d88a06242e77df5360" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 33291 } } }
I20260926 03:43:53.993580 2419 tablet_replica.cc:442] T fffd1e2b87f2458eba56a1d319da8706 P df904b94fdc14a05b7d78665d955f86f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 03:43:53.995226 2442 tablet_replica.cc:439] T fffd1e2b87f2458eba56a1d319da8706 P 88b981e00e3e43089b67f936a8e86e51: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "88b981e00e3e43089b67f936a8e86e51" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df904b94fdc14a05b7d78665d955f86f" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 33137 } } peers { permanent_uuid: "88b981e00e3e43089b67f936a8e86e51" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 46757 } } peers { permanent_uuid: "cdbcd498e18240d88a06242e77df5360" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 33291 } } }
I20260926 03:43:53.995565 2442 tablet_replica.cc:442] T fffd1e2b87f2458eba56a1d319da8706 P 88b981e00e3e43089b67f936a8e86e51: This TxnStatusTablet replica's current role is: LEADER
I20260926 03:43:54.007457 2431 tablet_replica.cc:439] T fffd1e2b87f2458eba56a1d319da8706 P cdbcd498e18240d88a06242e77df5360: TxnStatusTablet state changed. Reason: New leader 88b981e00e3e43089b67f936a8e86e51. Latest consensus state: current_term: 1 leader_uuid: "88b981e00e3e43089b67f936a8e86e51" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df904b94fdc14a05b7d78665d955f86f" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 33137 } } peers { permanent_uuid: "88b981e00e3e43089b67f936a8e86e51" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 46757 } } peers { permanent_uuid: "cdbcd498e18240d88a06242e77df5360" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 33291 } } }
I20260926 03:43:54.007802 2431 tablet_replica.cc:442] T fffd1e2b87f2458eba56a1d319da8706 P cdbcd498e18240d88a06242e77df5360: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 03:43:54.011584 2442 tablet_replica.cc:439] T fffd1e2b87f2458eba56a1d319da8706 P 88b981e00e3e43089b67f936a8e86e51: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "88b981e00e3e43089b67f936a8e86e51" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df904b94fdc14a05b7d78665d955f86f" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 33137 } } peers { permanent_uuid: "88b981e00e3e43089b67f936a8e86e51" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 46757 } } peers { permanent_uuid: "cdbcd498e18240d88a06242e77df5360" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 33291 } } }
I20260926 03:43:54.011919 2442 tablet_replica.cc:442] T fffd1e2b87f2458eba56a1d319da8706 P 88b981e00e3e43089b67f936a8e86e51: This TxnStatusTablet replica's current role is: LEADER
I20260926 03:43:54.015766 2431 tablet_replica.cc:439] T fffd1e2b87f2458eba56a1d319da8706 P cdbcd498e18240d88a06242e77df5360: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "88b981e00e3e43089b67f936a8e86e51" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df904b94fdc14a05b7d78665d955f86f" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 33137 } } peers { permanent_uuid: "88b981e00e3e43089b67f936a8e86e51" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 46757 } } peers { permanent_uuid: "cdbcd498e18240d88a06242e77df5360" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 33291 } } }
I20260926 03:43:54.016014 2431 tablet_replica.cc:442] T fffd1e2b87f2458eba56a1d319da8706 P cdbcd498e18240d88a06242e77df5360: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 03:43:54.027084 2457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:54.008354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44238 (local address 127.0.194.194:46757)
0926 03:43:54.008633 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0926 03:43:54.008643 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 03:43:54.017410 (+ 8767us) server_negotiation.cc:407] Connection header received
0926 03:43:54.017712 (+ 302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:43:54.017720 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:43:54.017805 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:43:54.017869 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:43:54.018757 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:54.019982 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:54.024808 (+ 4826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:54.025504 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:54.025643 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:43:54.026240 (+ 597us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:43:54.026378 (+ 138us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:43:54.026472 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0926 03:43:54.026622 (+ 150us) server_negotiation.cc:299] Negotiation successful
0926 03:43:54.026707 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260926 03:43:54.046631 2464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:54.039239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.194:46757 (local address 127.0.194.194:42291)
0926 03:43:54.039928 (+ 689us) negotiation.cc:107] Waiting for socket to connect
0926 03:43:54.039953 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0926 03:43:54.040091 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:43:54.040451 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:43:54.040456 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:43:54.040467 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 03:43:54.040827 (+ 360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:43:54.040838 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:43:54.042327 (+ 1489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:43:54.042333 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:43:54.044243 (+ 1910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:43:54.044258 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:43:54.045311 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:43:54.045318 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:43:54.045559 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:43:54.045637 (+ 78us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 03:43:54.046078 (+ 441us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 03:43:54.046087 (+ 9us) client_negotiation.cc:769] Sending connection context
0926 03:43:54.046155 (+ 68us) client_negotiation.cc:240] Negotiation successful
0926 03:43:54.046265 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":458,"threads_started":1}
I20260926 03:43:54.046651 2457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:54.039500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.194:42291 (local address 127.0.194.194:46757)
0926 03:43:54.039696 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0926 03:43:54.039702 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:43:54.040041 (+ 339us) server_negotiation.cc:407] Connection header received
0926 03:43:54.040215 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:43:54.040221 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:43:54.040328 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:43:54.040429 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:43:54.040993 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:54.042188 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:54.044421 (+ 2233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:54.045095 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:54.045241 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:43:54.045832 (+ 591us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:43:54.045971 (+ 139us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:43:54.046068 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0926 03:43:54.046201 (+ 133us) server_negotiation.cc:299] Negotiation successful
0926 03:43:54.046278 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 03:43:54.056363 2345 raft_consensus.cc:1261] T af4366574be04591b6ada84acdf05de3 P cdbcd498e18240d88a06242e77df5360 [term 1 FOLLOWER]: Refusing update from remote peer 88b981e00e3e43089b67f936a8e86e51: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:43:54.056442 2063 raft_consensus.cc:1261] T af4366574be04591b6ada84acdf05de3 P df904b94fdc14a05b7d78665d955f86f [term 1 FOLLOWER]: Refusing update from remote peer 88b981e00e3e43089b67f936a8e86e51: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:43:54.056962 2417 consensus_queue.cc:1035] T af4366574be04591b6ada84acdf05de3 P 88b981e00e3e43089b67f936a8e86e51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdbcd498e18240d88a06242e77df5360" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 33291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:43:54.057308 2442 consensus_queue.cc:1035] T af4366574be04591b6ada84acdf05de3 P 88b981e00e3e43089b67f936a8e86e51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df904b94fdc14a05b7d78665d955f86f" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 33137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:43:54.067430 2424 mvcc.cc:205] Tried to move back new op lower bound from 7333454782688796672 to 7333454781240111104. Current Snapshot: MvccSnapshot[applied={T|T < 7333454782688796672}]
I20260926 03:43:54.118168 1905 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53016:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:43:54.118373 1905 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53016:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:43:54.121644 1905 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5908aa01446f46239684277b300e638a: Sending DeleteTablet for 3 replicas of tablet af4366574be04591b6ada84acdf05de3
I20260926 03:43:54.122468 2044 tablet_service.cc:1514] Processing DeleteTablet for tablet af4366574be04591b6ada84acdf05de3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:43:54 UTC) from {username='kudu', principal='kudu/127.0.194.254@KRBTEST.COM'} at 127.0.0.1:36392
I20260926 03:43:54.122524 2182 tablet_service.cc:1514] Processing DeleteTablet for tablet af4366574be04591b6ada84acdf05de3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:43:54 UTC) from {username='kudu', principal='kudu/127.0.194.254@KRBTEST.COM'} at 127.0.0.1:44226
I20260926 03:43:54.122820 2321 tablet_service.cc:1514] Processing DeleteTablet for tablet af4366574be04591b6ada84acdf05de3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:43:54 UTC) from {username='kudu', principal='kudu/127.0.194.254@KRBTEST.COM'} at 127.0.0.1:36772
I20260926 03:43:54.123118 2469 tablet_replica.cc:330] T af4366574be04591b6ada84acdf05de3 P df904b94fdc14a05b7d78665d955f86f: stopping tablet replica
I20260926 03:43:54.123332 2469 raft_consensus.cc:2229] T af4366574be04591b6ada84acdf05de3 P df904b94fdc14a05b7d78665d955f86f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:43:54.123430 2470 tablet_replica.cc:330] T af4366574be04591b6ada84acdf05de3 P 88b981e00e3e43089b67f936a8e86e51: stopping tablet replica
I20260926 03:43:54.123596 2469 raft_consensus.cc:2258] T af4366574be04591b6ada84acdf05de3 P df904b94fdc14a05b7d78665d955f86f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:43:54.123600 2470 raft_consensus.cc:2229] T af4366574be04591b6ada84acdf05de3 P 88b981e00e3e43089b67f936a8e86e51 [term 1 LEADER]: Raft consensus shutting down.
I20260926 03:43:54.124202 2470 raft_consensus.cc:2258] T af4366574be04591b6ada84acdf05de3 P 88b981e00e3e43089b67f936a8e86e51 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:43:54.124732 2469 ts_tablet_manager.cc:1905] T af4366574be04591b6ada84acdf05de3 P df904b94fdc14a05b7d78665d955f86f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:43:54.125275 2470 ts_tablet_manager.cc:1905] T af4366574be04591b6ada84acdf05de3 P 88b981e00e3e43089b67f936a8e86e51: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:43:54.131767 2469 ts_tablet_manager.cc:1918] T af4366574be04591b6ada84acdf05de3 P df904b94fdc14a05b7d78665d955f86f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 03:43:54.131906 2469 log.cc:1196] T af4366574be04591b6ada84acdf05de3 P df904b94fdc14a05b7d78665d955f86f: Deleting WAL directory at /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790394221564982-779-0/minicluster-data/ts-0/wal/wals/af4366574be04591b6ada84acdf05de3
I20260926 03:43:54.132342 2469 ts_tablet_manager.cc:1939] T af4366574be04591b6ada84acdf05de3 P df904b94fdc14a05b7d78665d955f86f: Deleting consensus metadata
I20260926 03:43:54.132392 2470 ts_tablet_manager.cc:1918] T af4366574be04591b6ada84acdf05de3 P 88b981e00e3e43089b67f936a8e86e51: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 03:43:54.132519 2470 log.cc:1196] T af4366574be04591b6ada84acdf05de3 P 88b981e00e3e43089b67f936a8e86e51: Deleting WAL directory at /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790394221564982-779-0/minicluster-data/ts-1/wal/wals/af4366574be04591b6ada84acdf05de3
I20260926 03:43:54.132977 2470 ts_tablet_manager.cc:1939] T af4366574be04591b6ada84acdf05de3 P 88b981e00e3e43089b67f936a8e86e51: Deleting consensus metadata
I20260926 03:43:54.133566 1893 catalog_manager.cc:4841] TS df904b94fdc14a05b7d78665d955f86f (127.0.194.193:33137): tablet af4366574be04591b6ada84acdf05de3 (table test-table [id=ea0d984049ef42dcabb201d6e2d69461]) successfully deleted
I20260926 03:43:54.133989 1892 catalog_manager.cc:4841] TS 88b981e00e3e43089b67f936a8e86e51 (127.0.194.194:46757): tablet af4366574be04591b6ada84acdf05de3 (table test-table [id=ea0d984049ef42dcabb201d6e2d69461]) successfully deleted
I20260926 03:43:54.135847 2471 tablet_replica.cc:330] T af4366574be04591b6ada84acdf05de3 P cdbcd498e18240d88a06242e77df5360: stopping tablet replica
I20260926 03:43:54.136052 2471 raft_consensus.cc:2229] T af4366574be04591b6ada84acdf05de3 P cdbcd498e18240d88a06242e77df5360 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:43:54.136309 2471 raft_consensus.cc:2258] T af4366574be04591b6ada84acdf05de3 P cdbcd498e18240d88a06242e77df5360 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:43:54.137333 2471 ts_tablet_manager.cc:1905] T af4366574be04591b6ada84acdf05de3 P cdbcd498e18240d88a06242e77df5360: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:43:54.144313 2471 ts_tablet_manager.cc:1918] T af4366574be04591b6ada84acdf05de3 P cdbcd498e18240d88a06242e77df5360: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 03:43:54.144590 2471 log.cc:1196] T af4366574be04591b6ada84acdf05de3 P cdbcd498e18240d88a06242e77df5360: Deleting WAL directory at /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790394221564982-779-0/minicluster-data/ts-2/wal/wals/af4366574be04591b6ada84acdf05de3
I20260926 03:43:54.145033 2471 ts_tablet_manager.cc:1939] T af4366574be04591b6ada84acdf05de3 P cdbcd498e18240d88a06242e77df5360: Deleting consensus metadata
I20260926 03:43:54.146245 1891 catalog_manager.cc:4841] TS cdbcd498e18240d88a06242e77df5360 (127.0.194.195:33291): tablet af4366574be04591b6ada84acdf05de3 (table test-table [id=ea0d984049ef42dcabb201d6e2d69461]) successfully deleted
Sep 26 03:43:54 dist-test-slave-n2th krb5kdc[1862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394233, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.194.193@KRBTEST.COM
I20260926 03:43:54.153851 2409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:54.135958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36394 (local address 127.0.194.193:33137)
0926 03:43:54.136156 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0926 03:43:54.136164 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:43:54.139370 (+ 3206us) server_negotiation.cc:407] Connection header received
0926 03:43:54.139584 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:43:54.139588 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:43:54.139660 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:43:54.139767 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:43:54.141163 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:54.141856 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:54.143152 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:54.143592 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:54.143731 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:43:54.148064 (+ 4333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:43:54.148076 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:43:54.148092 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:43:54.148127 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:43:54.151115 (+ 2988us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:43:54.152013 (+ 898us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:43:54.152020 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:43:54.152022 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:43:54.152147 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:43:54.152553 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:43:54.152556 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:43:54.152558 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:43:54.152853 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:43:54.153073 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0926 03:43:54.153392 (+ 319us) server_negotiation.cc:299] Negotiation successful
0926 03:43:54.153570 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
W20260926 03:43:54.154644 2109 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36394
I20260926 03:43:54.170972 2002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:54.157536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53028 (local address 127.0.194.254:37875)
0926 03:43:54.157747 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0926 03:43:54.157754 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:43:54.158228 (+ 474us) server_negotiation.cc:407] Connection header received
0926 03:43:54.158360 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:43:54.158366 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:43:54.158444 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:43:54.158552 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:43:54.159797 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:54.160837 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:54.162099 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:54.162681 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:54.162883 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:43:54.164992 (+ 2109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:43:54.164999 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:43:54.165005 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:43:54.165042 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:43:54.168453 (+ 3411us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:43:54.169048 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:43:54.169055 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:43:54.169059 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:43:54.169209 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:43:54.169527 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:43:54.169532 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:43:54.169535 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:43:54.169878 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:43:54.170137 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0926 03:43:54.170366 (+ 229us) server_negotiation.cc:299] Negotiation successful
0926 03:43:54.170581 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
W20260926 03:43:54.171702 1905 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53028
I20260926 03:43:54.173894 779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu pid 1978
I20260926 03:43:54.187690 2409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:54.174842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36406 (local address 127.0.194.193:33137)
0926 03:43:54.175045 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0926 03:43:54.175050 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:43:54.175557 (+ 507us) server_negotiation.cc:407] Connection header received
0926 03:43:54.175752 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:43:54.175756 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:43:54.175817 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:43:54.175925 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:43:54.176985 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:54.177702 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:54.178950 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:54.179332 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:54.179465 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:43:54.181651 (+ 2186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:43:54.181655 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:43:54.181659 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:43:54.181688 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:43:54.184591 (+ 2903us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:43:54.185455 (+ 864us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:43:54.185463 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:43:54.185470 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:43:54.185608 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:43:54.185993 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:43:54.186000 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:43:54.186004 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:43:54.186312 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:43:54.186588 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0926 03:43:54.187101 (+ 513us) server_negotiation.cc:299] Negotiation successful
0926 03:43:54.187315 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260926 03:43:54.188313 2109 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36406
W20260926 03:43:54.188872 779 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu pid 1978: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 03:43:54.189065 779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu with pid 1978
I20260926 03:43:54.209617 779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu pid 2118
Sep 26 03:43:54 dist-test-slave-n2th krb5kdc[1862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394233, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.194.194@KRBTEST.COM
I20260926 03:43:54.227375 2457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:54.210442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44248 (local address 127.0.194.194:46757)
0926 03:43:54.210686 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0926 03:43:54.210692 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:43:54.210771 (+ 79us) server_negotiation.cc:407] Connection header received
0926 03:43:54.210841 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:43:54.210848 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:43:54.210934 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:43:54.211032 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:43:54.212136 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:54.213276 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:54.214689 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:54.215421 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:54.215553 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:43:54.220005 (+ 4452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:43:54.220018 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:43:54.220035 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:43:54.220076 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:43:54.224203 (+ 4127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:43:54.225108 (+ 905us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:43:54.225117 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:43:54.225121 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:43:54.225274 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:43:54.225747 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:43:54.225753 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:43:54.225756 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:43:54.226197 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:43:54.226564 (+ 367us) server_negotiation.cc:1035] Waiting for connection context
0926 03:43:54.226831 (+ 267us) server_negotiation.cc:299] Negotiation successful
0926 03:43:54.227034 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
W20260926 03:43:54.228120 2248 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44248
W20260926 03:43:54.228737 779 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu pid 2118: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 03:43:54.228912 779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu with pid 2118
I20260926 03:43:54.247823 779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu pid 2257
Sep 26 03:43:54 dist-test-slave-n2th krb5kdc[1862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394233, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.194.195@KRBTEST.COM
I20260926 03:43:54.262633 2477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:54.248653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36792 (local address 127.0.194.195:33291)
0926 03:43:54.249224 (+ 571us) server_negotiation.cc:206] Beginning negotiation
0926 03:43:54.249228 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:43:54.249243 (+ 15us) server_negotiation.cc:407] Connection header received
0926 03:43:54.249291 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:43:54.249295 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:43:54.249348 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:43:54.249468 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:43:54.250699 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:54.251592 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:54.252875 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:54.253322 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:54.253462 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:43:54.257700 (+ 4238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:43:54.257714 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:43:54.257729 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:43:54.257762 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:43:54.260382 (+ 2620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:43:54.260958 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:43:54.260965 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:43:54.260967 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:43:54.261082 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:43:54.261381 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:43:54.261384 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:43:54.261386 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:43:54.261695 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:43:54.261911 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0926 03:43:54.262121 (+ 210us) server_negotiation.cc:299] Negotiation successful
0926 03:43:54.262273 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":467,"thread_start_us":285,"threads_started":1}
W20260926 03:43:54.263346 2387 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36792
W20260926 03:43:54.264021 779 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu pid 2257: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 03:43:54.264176 779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu with pid 2257
I20260926 03:43:54.285553 779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu pid 1875
I20260926 03:43:54.286695 1947 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:43:54.401194 779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu with pid 1875
2026-09-26T03:43:54Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (4525 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790394221564982-779-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:43:54 dist-test-slave-n2th krb5kdc[2481](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:43:54 dist-test-slave-n2th krb5kdc[2481](info): set up 2 sockets
Sep 26 03:43:54 dist-test-slave-n2th krb5kdc[2481](info): commencing operation
krb5kdc: starting...
W20260926 03:43:55.494838 779 mini_kdc.cc:121] Time spent starting KDC: real 1.050s user 0.002s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790394221564982-779-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790394221564982-779-0/krb5kdc/test-user.keytab.
Sep 26 03:43:55 dist-test-slave-n2th krb5kdc[2481](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394235, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:43:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:43:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.194.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.194.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.194.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790394221564982-779-0/krb5kdc/kudu_127.0.194.254.keytab.
Entry for principal kudu/127.0.194.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790394221564982-779-0/krb5kdc/kudu_127.0.194.254.keytab.
I20260926 03:43:55.685931 779 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
/tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790394221564982-779-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790394221564982-779-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790394221564982-779-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790394221564982-779-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.194.254:43979
--webserver_interface=127.0.194.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790394221564982-779-0/krb5kdc/kudu_127.0.194.254.keytab
--principal=kudu/127.0.194.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.194.212:38377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.194.254:43979
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790394221564982-779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790394221564982-779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790394221564982-779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:43:56.015537 2494 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:43:56.015844 2494 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:43:56.015933 2494 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:43:56.023177 2494 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:43:56.023286 2494 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260926 03:43:56.023348 2494 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:43:56.023412 2494 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:43:56.023460 2494 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:43:56.023514 2494 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:43:56.033409 2494 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.194.212:38377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790394221564982-779-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790394221564982-779-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.194.254:43979
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790394221564982-779-0/krb5kdc/kudu_127.0.194.254.keytab
--principal=kudu/127.0.194.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.194.254:43979
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790394221564982-779-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.194.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790394221564982-779-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:43:56.034667 2494 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:43:56.035979 2494 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:43:56.047860 2499 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:43:56.047823 2500 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:43:56.047781 2502 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:43:56.048816 2494 server_base.cc:1034] running on GCE node
I20260926 03:43:56.049714 2494 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:43:56.051265 2494 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:43:56.052515 2494 hybrid_clock.cc:648] HybridClock initialized: now 1790394236052445 us; error 48 us; skew 500 ppm
I20260926 03:43:56.052855 2494 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:43:56 dist-test-slave-n2th krb5kdc[2481](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394236, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:43:56.056111 2494 init.cc:377] Logged in from keytab as kudu/127.0.194.254@KRBTEST.COM (short username kudu)
I20260926 03:43:56.057770 2494 webserver.cc:460] Webserver started at http://127.0.194.254:37113/ using document root <none> and password file <none>
I20260926 03:43:56.058506 2494 fs_manager.cc:359] Metadata directory not provided
I20260926 03:43:56.058637 2494 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:43:56.059022 2494 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:43:56.062054 2494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790394221564982-779-0/minicluster-data/master-0/data/instance:
uuid: "0ccb5168dad34eb2a79efa733da1e3f6"
format_stamp: "Formatted at 2026-09-26 03:43:56 on dist-test-slave-n2th"
server_key: "3fe94f13590df169523903dd1f87ca43"
server_key_iv: "dd59da5c1a035f2186cc08439bed4543"
server_key_version: "encryptionkey@0"
I20260926 03:43:56.062934 2494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790394221564982-779-0/minicluster-data/master-0/wal/instance:
uuid: "0ccb5168dad34eb2a79efa733da1e3f6"
format_stamp: "Formatted at 2026-09-26 03:43:56 on dist-test-slave-n2th"
server_key: "3fe94f13590df169523903dd1f87ca43"
server_key_iv: "dd59da5c1a035f2186cc08439bed4543"
server_key_version: "encryptionkey@0"
I20260926 03:43:56.068193 2494 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260926 03:43:56.071750 2509 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:43:56.073094 2494 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 03:43:56.073295 2494 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790394221564982-779-0/minicluster-data/master-0/data,/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790394221564982-779-0/minicluster-data/master-0/wal
uuid: "0ccb5168dad34eb2a79efa733da1e3f6"
format_stamp: "Formatted at 2026-09-26 03:43:56 on dist-test-slave-n2th"
server_key: "3fe94f13590df169523903dd1f87ca43"
server_key_iv: "dd59da5c1a035f2186cc08439bed4543"
server_key_version: "encryptionkey@0"
I20260926 03:43:56.073527 2494 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790394221564982-779-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790394221564982-779-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790394221564982-779-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:43:56.096446 2494 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:43:56.100765 2494 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:43:56.101054 2494 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:43:56.123404 2494 rpc_server.cc:307] RPC server started. Bound to: 127.0.194.254:43979
I20260926 03:43:56.123411 2570 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.194.254:43979 every 8 connection(s)
I20260926 03:43:56.124809 2494 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790394221564982-779-0/minicluster-data/master-0/data/info.pb
I20260926 03:43:56.128690 2571 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:43:56.135252 779 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu as pid 2494
I20260926 03:43:56.135490 779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790394221564982-779-0/minicluster-data/master-0/wal/instance
I20260926 03:43:56.136313 779 external_mini_cluster.cc:1428] Setting key 15c365397327db43781329f735ade069
I20260926 03:43:56.140430 2571 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ccb5168dad34eb2a79efa733da1e3f6: Bootstrap starting.
I20260926 03:43:56.144328 2571 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0ccb5168dad34eb2a79efa733da1e3f6: Neither blocks nor log segments found. Creating new log.
I20260926 03:43:56.145402 2571 log.cc:824] T 00000000000000000000000000000000 P 0ccb5168dad34eb2a79efa733da1e3f6: Log is configured to *not* fsync() on all Append() calls
I20260926 03:43:56.148648 2571 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ccb5168dad34eb2a79efa733da1e3f6: No bootstrap required, opened a new log
Sep 26 03:43:56 dist-test-slave-n2th krb5kdc[2481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394235, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.194.254@KRBTEST.COM
I20260926 03:43:56.159569 2571 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0ccb5168dad34eb2a79efa733da1e3f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ccb5168dad34eb2a79efa733da1e3f6" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 43979 } }
I20260926 03:43:56.160075 2571 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0ccb5168dad34eb2a79efa733da1e3f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:43:56.160192 2571 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0ccb5168dad34eb2a79efa733da1e3f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ccb5168dad34eb2a79efa733da1e3f6, State: Initialized, Role: FOLLOWER
I20260926 03:43:56.160897 2571 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0ccb5168dad34eb2a79efa733da1e3f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ccb5168dad34eb2a79efa733da1e3f6" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 43979 } }
I20260926 03:43:56.161186 2571 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0ccb5168dad34eb2a79efa733da1e3f6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:43:56.161334 2571 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0ccb5168dad34eb2a79efa733da1e3f6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:43:56.161517 2571 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0ccb5168dad34eb2a79efa733da1e3f6 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:43:56.161674 2574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:56.138527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52858 (local address 127.0.194.254:43979)
0926 03:43:56.139431 (+ 904us) server_negotiation.cc:206] Beginning negotiation
0926 03:43:56.139445 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 03:43:56.139488 (+ 43us) server_negotiation.cc:407] Connection header received
0926 03:43:56.141242 (+ 1754us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:43:56.141265 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:43:56.141685 (+ 420us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:43:56.142052 (+ 367us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:43:56.143932 (+ 1880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:56.145333 (+ 1401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:56.146906 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:56.147576 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:56.147707 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:43:56.152436 (+ 4729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:43:56.152448 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:43:56.152471 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:43:56.152526 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:43:56.156787 (+ 4261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:43:56.157583 (+ 796us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:43:56.157594 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:43:56.157605 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:43:56.157856 (+ 251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:43:56.158223 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:43:56.158228 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:43:56.158231 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:43:56.159027 (+ 796us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:43:56.159348 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0926 03:43:56.159689 (+ 341us) server_negotiation.cc:299] Negotiation successful
0926 03:43:56.160196 (+ 507us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":688,"thread_start_us":466,"threads_started":1}
I20260926 03:43:56.164098 2571 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0ccb5168dad34eb2a79efa733da1e3f6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ccb5168dad34eb2a79efa733da1e3f6" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 43979 } }
I20260926 03:43:56.164754 2571 leader_election.cc:302] T 00000000000000000000000000000000 P 0ccb5168dad34eb2a79efa733da1e3f6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0ccb5168dad34eb2a79efa733da1e3f6; no voters:
I20260926 03:43:56.165338 2571 leader_election.cc:288] T 00000000000000000000000000000000 P 0ccb5168dad34eb2a79efa733da1e3f6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:43:56.165493 2576 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0ccb5168dad34eb2a79efa733da1e3f6 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:43:56.166217 2576 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0ccb5168dad34eb2a79efa733da1e3f6 [term 1 LEADER]: Becoming Leader. State: Replica: 0ccb5168dad34eb2a79efa733da1e3f6, State: Running, Role: LEADER
I20260926 03:43:56.166950 2576 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0ccb5168dad34eb2a79efa733da1e3f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ccb5168dad34eb2a79efa733da1e3f6" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 43979 } }
I20260926 03:43:56.167582 2571 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0ccb5168dad34eb2a79efa733da1e3f6 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:43:56.169792 2577 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ccb5168dad34eb2a79efa733da1e3f6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0ccb5168dad34eb2a79efa733da1e3f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ccb5168dad34eb2a79efa733da1e3f6" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 43979 } } }
I20260926 03:43:56.170289 2578 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ccb5168dad34eb2a79efa733da1e3f6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0ccb5168dad34eb2a79efa733da1e3f6. Latest consensus state: current_term: 1 leader_uuid: "0ccb5168dad34eb2a79efa733da1e3f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ccb5168dad34eb2a79efa733da1e3f6" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 43979 } } }
I20260926 03:43:56.170533 2577 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ccb5168dad34eb2a79efa733da1e3f6 [sys.catalog]: This master's current role is: LEADER
I20260926 03:43:56.170583 2578 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ccb5168dad34eb2a79efa733da1e3f6 [sys.catalog]: This master's current role is: LEADER
I20260926 03:43:56.175530 2584 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:43:56.181419 2584 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:43:56.190935 2584 catalog_manager.cc:1347] Generated new cluster ID: 316297c5b11a4aa2a8871949293f531d
I20260926 03:43:56.191072 2584 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:43:56.214212 2584 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:43:56.215690 2584 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:43:56.232331 2584 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0ccb5168dad34eb2a79efa733da1e3f6: Generated new TSK 0
I20260926 03:43:56.233237 2584 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.194.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.194.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.194.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790394221564982-779-0/krb5kdc/kudu_127.0.194.193.keytab.
Entry for principal kudu/127.0.194.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790394221564982-779-0/krb5kdc/kudu_127.0.194.193.keytab.
I20260926 03:43:56.293756 779 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
/tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790394221564982-779-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790394221564982-779-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790394221564982-779-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790394221564982-779-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.194.193:0
--local_ip_for_outbound_sockets=127.0.194.193
--webserver_interface=127.0.194.193
--webserver_port=0
--tserver_master_addrs=127.0.194.254:43979
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790394221564982-779-0/krb5kdc/kudu_127.0.194.193.keytab
--principal=kudu/127.0.194.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.194.212:38377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790394221564982-779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790394221564982-779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790394221564982-779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:43:56.646337 2597 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:43:56.646620 2597 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:43:56.646836 2597 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:43:56.654177 2597 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:43:56.654284 2597 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:43:56.654479 2597 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260926 03:43:56.654562 2597 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 03:43:56.654685 2597 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.194.193
I20260926 03:43:56.664417 2597 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.194.212:38377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790394221564982-779-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790394221564982-779-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790394221564982-779-0/krb5kdc/kudu_127.0.194.193.keytab
--principal=kudu/127.0.194.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.194.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790394221564982-779-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.194.193
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.194.254:43979
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.194.193
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790394221564982-779-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:43:56.665714 2597 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:43:56.667088 2597 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:43:56.680622 2602 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:43:56.680563 2605 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:43:56.680617 2603 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:43:56.681576 2597 server_base.cc:1034] running on GCE node
I20260926 03:43:56.682192 2597 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:43:56.683236 2597 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:43:56.684432 2597 hybrid_clock.cc:648] HybridClock initialized: now 1790394236684365 us; error 57 us; skew 500 ppm
I20260926 03:43:56.684752 2597 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:43:56 dist-test-slave-n2th krb5kdc[2481](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394236, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:43:56.687992 2597 init.cc:377] Logged in from keytab as kudu/127.0.194.193@KRBTEST.COM (short username kudu)
I20260926 03:43:56.689443 2597 webserver.cc:460] Webserver started at http://127.0.194.193:37039/ using document root <none> and password file <none>
I20260926 03:43:56.690142 2597 fs_manager.cc:359] Metadata directory not provided
I20260926 03:43:56.690276 2597 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:43:56.690604 2597 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:43:56.693571 2597 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790394221564982-779-0/minicluster-data/ts-0/data/instance:
uuid: "f496bcbb6d8644ad962eceeeac3f82ca"
format_stamp: "Formatted at 2026-09-26 03:43:56 on dist-test-slave-n2th"
server_key: "5e7286fd94f92addae3dddfbfafd7b03"
server_key_iv: "d2e7dec4d65aadd37b766bea1c3a0dd1"
server_key_version: "encryptionkey@0"
I20260926 03:43:56.694380 2597 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790394221564982-779-0/minicluster-data/ts-0/wal/instance:
uuid: "f496bcbb6d8644ad962eceeeac3f82ca"
format_stamp: "Formatted at 2026-09-26 03:43:56 on dist-test-slave-n2th"
server_key: "5e7286fd94f92addae3dddfbfafd7b03"
server_key_iv: "d2e7dec4d65aadd37b766bea1c3a0dd1"
server_key_version: "encryptionkey@0"
I20260926 03:43:56.699900 2597 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260926 03:43:56.703522 2612 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:43:56.705700 2597 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260926 03:43:56.705894 2597 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790394221564982-779-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790394221564982-779-0/minicluster-data/ts-0/wal
uuid: "f496bcbb6d8644ad962eceeeac3f82ca"
format_stamp: "Formatted at 2026-09-26 03:43:56 on dist-test-slave-n2th"
server_key: "5e7286fd94f92addae3dddfbfafd7b03"
server_key_iv: "d2e7dec4d65aadd37b766bea1c3a0dd1"
server_key_version: "encryptionkey@0"
I20260926 03:43:56.706146 2597 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790394221564982-779-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790394221564982-779-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790394221564982-779-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:43:56.724340 2597 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:43:56.728447 2597 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:43:56.728762 2597 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:43:56.731587 2597 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:43:56.731710 2597 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:43:56.731868 2597 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:43:56.731966 2597 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 03:43:56 dist-test-slave-n2th krb5kdc[2481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394236, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.193@KRBTEST.COM for kudu/127.0.194.254@KRBTEST.COM
I20260926 03:43:56.757063 2621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:56.732209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.193:51033 (local address 127.0.194.254:43979)
0926 03:43:56.732862 (+ 653us) server_negotiation.cc:206] Beginning negotiation
0926 03:43:56.732867 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:43:56.733899 (+ 1032us) server_negotiation.cc:407] Connection header received
0926 03:43:56.736212 (+ 2313us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:43:56.736217 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:43:56.736311 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:43:56.736437 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:43:56.738682 (+ 2245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:56.739636 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:56.743183 (+ 3547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:56.743769 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:56.743920 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:43:56.749174 (+ 5254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:43:56.749185 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:43:56.749194 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:43:56.749254 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:43:56.753471 (+ 4217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:43:56.754578 (+ 1107us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:43:56.754583 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:43:56.754585 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:43:56.754709 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:43:56.755198 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:43:56.755206 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:43:56.755211 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:43:56.755596 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:43:56.755745 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0926 03:43:56.756476 (+ 731us) server_negotiation.cc:299] Negotiation successful
0926 03:43:56.756694 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":543,"thread_start_us":442,"threads_started":1}
I20260926 03:43:56.758139 2622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:56.732873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:43979 (local address 127.0.194.193:51033)
0926 03:43:56.733723 (+ 850us) negotiation.cc:107] Waiting for socket to connect
0926 03:43:56.733762 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 03:43:56.735904 (+ 2142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:43:56.736692 (+ 788us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:43:56.736700 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:43:56.737527 (+ 827us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:43:56.738492 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:43:56.738514 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:43:56.739843 (+ 1329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:43:56.739849 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:43:56.742884 (+ 3035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:43:56.742910 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:43:56.744353 (+ 1443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:43:56.744364 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:43:56.744709 (+ 345us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:43:56.745663 (+ 954us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:43:56.745710 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:43:56.748867 (+ 3157us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:43:56.753659 (+ 4792us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:43:56.753675 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:43:56.753699 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:43:56.754428 (+ 729us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:43:56.754857 (+ 429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:43:56.754863 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:43:56.754867 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:43:56.755072 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:43:56.755738 (+ 666us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:43:56.755747 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:43:56.756104 (+ 357us) client_negotiation.cc:769] Sending connection context
0926 03:43:56.756380 (+ 276us) client_negotiation.cc:240] Negotiation successful
0926 03:43:56.756921 (+ 541us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":601,"thread_start_us":403,"threads_started":1}
I20260926 03:43:56.779765 2621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:56.764562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.193:55851 (local address 127.0.194.254:43979)
0926 03:43:56.764770 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0926 03:43:56.764778 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:43:56.765437 (+ 659us) server_negotiation.cc:407] Connection header received
0926 03:43:56.765652 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:43:56.765657 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:43:56.765732 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:43:56.765837 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:43:56.766837 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:56.767887 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:56.770466 (+ 2579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:56.770960 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:56.771094 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:43:56.772985 (+ 1891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:43:56.772990 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:43:56.772993 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:43:56.773024 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:43:56.776485 (+ 3461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:43:56.777526 (+ 1041us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:43:56.777530 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:43:56.777532 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:43:56.777657 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:43:56.778122 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:43:56.778126 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:43:56.778128 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:43:56.778549 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:43:56.778726 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0926 03:43:56.779312 (+ 586us) server_negotiation.cc:299] Negotiation successful
0926 03:43:56.779512 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 03:43:56.780347 2692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:56.764650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:43979 (local address 127.0.194.193:55851)
0926 03:43:56.765303 (+ 653us) negotiation.cc:107] Waiting for socket to connect
0926 03:43:56.765319 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 03:43:56.765509 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:43:56.765857 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:43:56.765862 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:43:56.766387 (+ 525us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:43:56.766656 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:43:56.766664 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:43:56.768071 (+ 1407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:43:56.768080 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:43:56.770297 (+ 2217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:43:56.770318 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:43:56.771168 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:43:56.771174 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:43:56.771332 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:43:56.771979 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:43:56.771997 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:43:56.772783 (+ 786us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:43:56.776634 (+ 3851us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:43:56.776640 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:43:56.776648 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:43:56.777354 (+ 706us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:43:56.777769 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:43:56.777776 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:43:56.777781 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:43:56.777965 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:43:56.778700 (+ 735us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:43:56.778706 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:43:56.778995 (+ 289us) client_negotiation.cc:769] Sending connection context
0926 03:43:56.779207 (+ 212us) client_negotiation.cc:240] Negotiation successful
0926 03:43:56.779670 (+ 463us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":360,"threads_started":1}
I20260926 03:43:56.789943 2597 rpc_server.cc:307] RPC server started. Bound to: 127.0.194.193:36187
I20260926 03:43:56.789989 2732 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.194.193:36187 every 8 connection(s)
I20260926 03:43:56.791260 2597 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790394221564982-779-0/minicluster-data/ts-0/data/info.pb
I20260926 03:43:56.793321 2733 heartbeater.cc:347] Connected to a master server at 127.0.194.254:43979
I20260926 03:43:56.793577 2733 heartbeater.cc:464] Registering TS with master...
I20260926 03:43:56.794251 2733 heartbeater.cc:511] Master 127.0.194.254:43979 requested a full tablet report, sending...
I20260926 03:43:56.795668 779 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu as pid 2597
I20260926 03:43:56.795850 779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790394221564982-779-0/minicluster-data/ts-0/wal/instance
I20260926 03:43:56.796470 779 external_mini_cluster.cc:1428] Setting key 7458acd7bed300f78417f7d1d0d75129
I20260926 03:43:56.796631 2526 ts_manager.cc:194] Registered new tserver with Master: f496bcbb6d8644ad962eceeeac3f82ca (127.0.194.193:36187)
I20260926 03:43:56.798811 2526 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.194.193@KRBTEST.COM'} at 127.0.194.193:51033
WARNING: no policy specified for kudu/127.0.194.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.194.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.194.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790394221564982-779-0/krb5kdc/kudu_127.0.194.194.keytab.
Entry for principal kudu/127.0.194.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790394221564982-779-0/krb5kdc/kudu_127.0.194.194.keytab.
I20260926 03:43:56.839637 779 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
/tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790394221564982-779-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790394221564982-779-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790394221564982-779-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790394221564982-779-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.194.194:0
--local_ip_for_outbound_sockets=127.0.194.194
--webserver_interface=127.0.194.194
--webserver_port=0
--tserver_master_addrs=127.0.194.254:43979
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790394221564982-779-0/krb5kdc/kudu_127.0.194.194.keytab
--principal=kudu/127.0.194.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.194.212:38377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790394221564982-779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790394221564982-779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790394221564982-779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:43:57.163296 2737 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:43:57.163656 2737 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:43:57.163837 2737 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:43:57.171365 2737 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:43:57.171530 2737 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:43:57.171694 2737 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260926 03:43:57.171782 2737 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 03:43:57.171898 2737 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.194.194
I20260926 03:43:57.182039 2737 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.194.212:38377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790394221564982-779-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790394221564982-779-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790394221564982-779-0/krb5kdc/kudu_127.0.194.194.keytab
--principal=kudu/127.0.194.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.194.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790394221564982-779-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.194.194
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.194.254:43979
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.194.194
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790394221564982-779-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:43:57.183403 2737 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:43:57.184762 2737 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:43:57.198155 2743 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:43:57.198225 2745 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:43:57.198247 2742 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:43:57.199398 2737 server_base.cc:1034] running on GCE node
I20260926 03:43:57.199962 2737 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:43:57.200881 2737 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:43:57.202073 2737 hybrid_clock.cc:648] HybridClock initialized: now 1790394237202029 us; error 45 us; skew 500 ppm
I20260926 03:43:57.202397 2737 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:43:57 dist-test-slave-n2th krb5kdc[2481](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394237, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:43:57.211376 2737 init.cc:377] Logged in from keytab as kudu/127.0.194.194@KRBTEST.COM (short username kudu)
I20260926 03:43:57.212980 2737 webserver.cc:460] Webserver started at http://127.0.194.194:42831/ using document root <none> and password file <none>
I20260926 03:43:57.213646 2737 fs_manager.cc:359] Metadata directory not provided
I20260926 03:43:57.213748 2737 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:43:57.214007 2737 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:43:57.216920 2737 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790394221564982-779-0/minicluster-data/ts-1/data/instance:
uuid: "f20d775ac36948528aadf2fb37a3ed11"
format_stamp: "Formatted at 2026-09-26 03:43:57 on dist-test-slave-n2th"
server_key: "9501cee22a46ffd7cfda9146de96adeb"
server_key_iv: "1190fc04a6b889c3cc3165ebda4fa7e1"
server_key_version: "encryptionkey@0"
I20260926 03:43:57.217651 2737 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790394221564982-779-0/minicluster-data/ts-1/wal/instance:
uuid: "f20d775ac36948528aadf2fb37a3ed11"
format_stamp: "Formatted at 2026-09-26 03:43:57 on dist-test-slave-n2th"
server_key: "9501cee22a46ffd7cfda9146de96adeb"
server_key_iv: "1190fc04a6b889c3cc3165ebda4fa7e1"
server_key_version: "encryptionkey@0"
I20260926 03:43:57.222920 2737 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260926 03:43:57.226277 2752 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:43:57.227718 2737 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 03:43:57.227897 2737 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790394221564982-779-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790394221564982-779-0/minicluster-data/ts-1/wal
uuid: "f20d775ac36948528aadf2fb37a3ed11"
format_stamp: "Formatted at 2026-09-26 03:43:57 on dist-test-slave-n2th"
server_key: "9501cee22a46ffd7cfda9146de96adeb"
server_key_iv: "1190fc04a6b889c3cc3165ebda4fa7e1"
server_key_version: "encryptionkey@0"
I20260926 03:43:57.228117 2737 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790394221564982-779-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790394221564982-779-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790394221564982-779-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:43:57.246508 2737 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:43:57.250699 2737 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:43:57.251048 2737 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:43:57.253767 2737 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:43:57.253909 2737 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:43:57.254065 2737 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:43:57.254163 2737 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 03:43:57 dist-test-slave-n2th krb5kdc[2481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394237, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.194@KRBTEST.COM for kudu/127.0.194.254@KRBTEST.COM
I20260926 03:43:57.278816 2621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:57.254212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.194:59239 (local address 127.0.194.254:43979)
0926 03:43:57.254401 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0926 03:43:57.254407 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:43:57.255952 (+ 1545us) server_negotiation.cc:407] Connection header received
0926 03:43:57.258364 (+ 2412us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:43:57.258374 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:43:57.258488 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:43:57.258620 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:43:57.261149 (+ 2529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:57.261911 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:57.264124 (+ 2213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:57.264740 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:57.264880 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:43:57.270815 (+ 5935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:43:57.270824 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:43:57.270831 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:43:57.270872 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:43:57.274978 (+ 4106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:43:57.276235 (+ 1257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:43:57.276239 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:43:57.276242 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:43:57.276370 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:43:57.276910 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:43:57.276916 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:43:57.276919 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:43:57.277216 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:43:57.277441 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0926 03:43:57.278174 (+ 733us) server_negotiation.cc:299] Negotiation successful
0926 03:43:57.278417 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 03:43:57.280025 2761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:57.254871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:43979 (local address 127.0.194.194:59239)
0926 03:43:57.255755 (+ 884us) negotiation.cc:107] Waiting for socket to connect
0926 03:43:57.255805 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 03:43:57.257878 (+ 2073us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:43:57.258924 (+ 1046us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:43:57.258932 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:43:57.259761 (+ 829us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:43:57.260861 (+ 1100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:43:57.260888 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:43:57.262062 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:43:57.262068 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:43:57.263923 (+ 1855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:43:57.263945 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:43:57.264986 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:43:57.264993 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:43:57.265240 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:43:57.266071 (+ 831us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:43:57.266112 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:43:57.270452 (+ 4340us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:43:57.275166 (+ 4714us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:43:57.275179 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:43:57.275208 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:43:57.276074 (+ 866us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:43:57.276507 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:43:57.276511 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:43:57.276515 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:43:57.276738 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:43:57.277346 (+ 608us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:43:57.277354 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:43:57.277805 (+ 451us) client_negotiation.cc:769] Sending connection context
0926 03:43:57.278084 (+ 279us) client_negotiation.cc:240] Negotiation successful
0926 03:43:57.278637 (+ 553us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":607,"thread_start_us":395,"threads_started":1}
I20260926 03:43:57.305390 2621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:57.286939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.194:54803 (local address 127.0.194.254:43979)
0926 03:43:57.287175 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0926 03:43:57.287182 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:43:57.288130 (+ 948us) server_negotiation.cc:407] Connection header received
0926 03:43:57.288404 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:43:57.288412 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:43:57.288552 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:43:57.288670 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:43:57.290274 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:57.291353 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:57.294087 (+ 2734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:57.294901 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:57.295064 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:43:57.297908 (+ 2844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:43:57.297913 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:43:57.297916 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:43:57.297950 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:43:57.301631 (+ 3681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:43:57.302843 (+ 1212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:43:57.302851 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:43:57.302858 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:43:57.303097 (+ 239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:43:57.303663 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:43:57.303670 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:43:57.303675 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:43:57.304182 (+ 507us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:43:57.304349 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0926 03:43:57.304922 (+ 573us) server_negotiation.cc:299] Negotiation successful
0926 03:43:57.305116 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 03:43:57.305806 2817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:57.286950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:43979 (local address 127.0.194.194:54803)
0926 03:43:57.287982 (+ 1032us) negotiation.cc:107] Waiting for socket to connect
0926 03:43:57.288008 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0926 03:43:57.288191 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:43:57.288736 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:43:57.288741 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:43:57.289481 (+ 740us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:43:57.290043 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:43:57.290059 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:43:57.291546 (+ 1487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:43:57.291556 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:43:57.293814 (+ 2258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:43:57.293841 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:43:57.295156 (+ 1315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:43:57.295169 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:43:57.295436 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:43:57.296310 (+ 874us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:43:57.296338 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:43:57.297691 (+ 1353us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:43:57.301838 (+ 4147us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:43:57.301845 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:43:57.301851 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:43:57.302652 (+ 801us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:43:57.303263 (+ 611us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:43:57.303270 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:43:57.303275 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:43:57.303524 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:43:57.304347 (+ 823us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:43:57.304355 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:43:57.304643 (+ 288us) client_negotiation.cc:769] Sending connection context
0926 03:43:57.304823 (+ 180us) client_negotiation.cc:240] Negotiation successful
0926 03:43:57.305189 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":823,"thread_start_us":622,"threads_started":1}
I20260926 03:43:57.306972 2737 rpc_server.cc:307] RPC server started. Bound to: 127.0.194.194:42655
I20260926 03:43:57.307070 2871 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.194.194:42655 every 8 connection(s)
I20260926 03:43:57.308527 2737 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790394221564982-779-0/minicluster-data/ts-1/data/info.pb
I20260926 03:43:57.310345 779 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu as pid 2737
I20260926 03:43:57.310547 779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790394221564982-779-0/minicluster-data/ts-1/wal/instance
I20260926 03:43:57.311141 779 external_mini_cluster.cc:1428] Setting key bf2be4c8006cd5fde5f0bb6cf4bc87c1
I20260926 03:43:57.311142 2872 heartbeater.cc:347] Connected to a master server at 127.0.194.254:43979
I20260926 03:43:57.311386 2872 heartbeater.cc:464] Registering TS with master...
I20260926 03:43:57.312223 2872 heartbeater.cc:511] Master 127.0.194.254:43979 requested a full tablet report, sending...
I20260926 03:43:57.313654 2526 ts_manager.cc:194] Registered new tserver with Master: f20d775ac36948528aadf2fb37a3ed11 (127.0.194.194:42655)
I20260926 03:43:57.314939 2526 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.194.194@KRBTEST.COM'} at 127.0.194.194:59239
WARNING: no policy specified for kudu/127.0.194.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.194.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.194.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790394221564982-779-0/krb5kdc/kudu_127.0.194.195.keytab.
Entry for principal kudu/127.0.194.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790394221564982-779-0/krb5kdc/kudu_127.0.194.195.keytab.
I20260926 03:43:57.352162 779 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
/tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790394221564982-779-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790394221564982-779-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790394221564982-779-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790394221564982-779-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.194.195:0
--local_ip_for_outbound_sockets=127.0.194.195
--webserver_interface=127.0.194.195
--webserver_port=0
--tserver_master_addrs=127.0.194.254:43979
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790394221564982-779-0/krb5kdc/kudu_127.0.194.195.keytab
--principal=kudu/127.0.194.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.194.212:38377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790394221564982-779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790394221564982-779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790394221564982-779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:43:57.676527 2876 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:43:57.676806 2876 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:43:57.676937 2876 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:43:57.684404 2876 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:43:57.684528 2876 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:43:57.684612 2876 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260926 03:43:57.684666 2876 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 03:43:57.684734 2876 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.194.195
I20260926 03:43:57.694638 2876 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.194.212:38377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790394221564982-779-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790394221564982-779-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790394221564982-779-0/krb5kdc/kudu_127.0.194.195.keytab
--principal=kudu/127.0.194.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.194.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790394221564982-779-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.194.195
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.194.254:43979
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.194.195
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790394221564982-779-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:43:57.696141 2876 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:43:57.697485 2876 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:43:57.710793 2881 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:43:57.710820 2882 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:43:57.711160 2884 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:43:57.711897 2876 server_base.cc:1034] running on GCE node
I20260926 03:43:57.712812 2876 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:43:57.716189 2876 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:43:57.717501 2876 hybrid_clock.cc:648] HybridClock initialized: now 1790394237717424 us; error 69 us; skew 500 ppm
I20260926 03:43:57.717852 2876 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:43:57 dist-test-slave-n2th krb5kdc[2481](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394237, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:43:57.722011 2876 init.cc:377] Logged in from keytab as kudu/127.0.194.195@KRBTEST.COM (short username kudu)
I20260926 03:43:57.723629 2876 webserver.cc:460] Webserver started at http://127.0.194.195:42893/ using document root <none> and password file <none>
I20260926 03:43:57.724361 2876 fs_manager.cc:359] Metadata directory not provided
I20260926 03:43:57.724514 2876 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:43:57.724845 2876 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:43:57.727914 2876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790394221564982-779-0/minicluster-data/ts-2/data/instance:
uuid: "737f7fce56574ef6a528cd2a48e8056d"
format_stamp: "Formatted at 2026-09-26 03:43:57 on dist-test-slave-n2th"
server_key: "e2f539a598c33ffbff665c5eb3e5cd83"
server_key_iv: "e7857d49fad0cda92a31c63c1a2280fe"
server_key_version: "encryptionkey@0"
I20260926 03:43:57.728726 2876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790394221564982-779-0/minicluster-data/ts-2/wal/instance:
uuid: "737f7fce56574ef6a528cd2a48e8056d"
format_stamp: "Formatted at 2026-09-26 03:43:57 on dist-test-slave-n2th"
server_key: "e2f539a598c33ffbff665c5eb3e5cd83"
server_key_iv: "e7857d49fad0cda92a31c63c1a2280fe"
server_key_version: "encryptionkey@0"
I20260926 03:43:57.734470 2876 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260926 03:43:57.738441 2891 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:43:57.740069 2876 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 03:43:57.740273 2876 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790394221564982-779-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790394221564982-779-0/minicluster-data/ts-2/wal
uuid: "737f7fce56574ef6a528cd2a48e8056d"
format_stamp: "Formatted at 2026-09-26 03:43:57 on dist-test-slave-n2th"
server_key: "e2f539a598c33ffbff665c5eb3e5cd83"
server_key_iv: "e7857d49fad0cda92a31c63c1a2280fe"
server_key_version: "encryptionkey@0"
I20260926 03:43:57.740499 2876 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790394221564982-779-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790394221564982-779-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790394221564982-779-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:43:57.769901 2876 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:43:57.774302 2876 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:43:57.774623 2876 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:43:57.777518 2876 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:43:57.777657 2876 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:43:57.777791 2876 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:43:57.777902 2876 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 03:43:57 dist-test-slave-n2th krb5kdc[2481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394237, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.195@KRBTEST.COM for kudu/127.0.194.254@KRBTEST.COM
I20260926 03:43:57.802436 2733 heartbeater.cc:503] Master 127.0.194.254:43979 was elected leader, sending a full tablet report...
I20260926 03:43:57.805460 2621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:57.778487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.195:39843 (local address 127.0.194.254:43979)
0926 03:43:57.778682 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0926 03:43:57.778689 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:43:57.780147 (+ 1458us) server_negotiation.cc:407] Connection header received
0926 03:43:57.783558 (+ 3411us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:43:57.783565 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:43:57.783663 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:43:57.783789 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:43:57.786032 (+ 2243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:57.787139 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:57.789736 (+ 2597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:57.790385 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:57.790518 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:43:57.796708 (+ 6190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:43:57.796719 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:43:57.796727 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:43:57.796820 (+ 93us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:43:57.801247 (+ 4427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:43:57.802507 (+ 1260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:43:57.802512 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:43:57.802516 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:43:57.802678 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:43:57.803345 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:43:57.803351 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:43:57.803353 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:43:57.803695 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:43:57.803851 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0926 03:43:57.804984 (+ 1133us) server_negotiation.cc:299] Negotiation successful
0926 03:43:57.805157 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 03:43:57.807773 2900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:57.779051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:43979 (local address 127.0.194.195:39843)
0926 03:43:57.779933 (+ 882us) negotiation.cc:107] Waiting for socket to connect
0926 03:43:57.779996 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0926 03:43:57.783126 (+ 3130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:43:57.784063 (+ 937us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:43:57.784072 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:43:57.784863 (+ 791us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:43:57.785829 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:43:57.785864 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:43:57.787322 (+ 1458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:43:57.787325 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:43:57.789541 (+ 2216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:43:57.789564 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:43:57.790969 (+ 1405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:43:57.790976 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:43:57.791282 (+ 306us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:43:57.792158 (+ 876us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:43:57.792193 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:43:57.796353 (+ 4160us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:43:57.801443 (+ 5090us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:43:57.801460 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:43:57.801487 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:43:57.802326 (+ 839us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:43:57.802831 (+ 505us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:43:57.802839 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:43:57.802845 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:43:57.803203 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:43:57.803861 (+ 658us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:43:57.803874 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:43:57.804484 (+ 610us) client_negotiation.cc:769] Sending connection context
0926 03:43:57.804883 (+ 399us) client_negotiation.cc:240] Negotiation successful
0926 03:43:57.805633 (+ 750us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":563,"thread_start_us":328,"threads_started":1}
I20260926 03:43:57.829630 2621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:57.814816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.195:54479 (local address 127.0.194.254:43979)
0926 03:43:57.815059 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0926 03:43:57.815069 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 03:43:57.815583 (+ 514us) server_negotiation.cc:407] Connection header received
0926 03:43:57.815741 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:43:57.815750 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:43:57.815865 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:43:57.816004 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:43:57.816885 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:57.818237 (+ 1352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:57.820328 (+ 2091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:57.820781 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:57.820927 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:43:57.822805 (+ 1878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:43:57.822812 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:43:57.822824 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:43:57.822865 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:43:57.826128 (+ 3263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:43:57.827244 (+ 1116us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:43:57.827249 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:43:57.827253 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:43:57.827443 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:43:57.827903 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:43:57.827908 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:43:57.827911 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:43:57.828337 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:43:57.828519 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0926 03:43:57.829028 (+ 509us) server_negotiation.cc:299] Negotiation successful
0926 03:43:57.829258 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 03:43:57.829843 2973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:57.814851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:43979 (local address 127.0.194.195:54479)
0926 03:43:57.815480 (+ 629us) negotiation.cc:107] Waiting for socket to connect
0926 03:43:57.815504 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 03:43:57.815619 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:43:57.816012 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:43:57.816015 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:43:57.816476 (+ 461us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:43:57.816740 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:43:57.816748 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:43:57.818395 (+ 1647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:43:57.818406 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:43:57.820177 (+ 1771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:43:57.820189 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:43:57.820991 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:43:57.820997 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:43:57.821163 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:43:57.821821 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:43:57.821839 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:43:57.822575 (+ 736us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:43:57.826283 (+ 3708us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:43:57.826290 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:43:57.826294 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:43:57.827097 (+ 803us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:43:57.827590 (+ 493us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:43:57.827600 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:43:57.827604 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:43:57.827772 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:43:57.828474 (+ 702us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:43:57.828482 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:43:57.828746 (+ 264us) client_negotiation.cc:769] Sending connection context
0926 03:43:57.828914 (+ 168us) client_negotiation.cc:240] Negotiation successful
0926 03:43:57.829285 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":341,"threads_started":1}
I20260926 03:43:57.835306 2876 rpc_server.cc:307] RPC server started. Bound to: 127.0.194.195:39703
I20260926 03:43:57.835340 3010 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.194.195:39703 every 8 connection(s)
I20260926 03:43:57.836572 2876 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790394221564982-779-0/minicluster-data/ts-2/data/info.pb
I20260926 03:43:57.838788 3011 heartbeater.cc:347] Connected to a master server at 127.0.194.254:43979
I20260926 03:43:57.839031 3011 heartbeater.cc:464] Registering TS with master...
I20260926 03:43:57.839715 3011 heartbeater.cc:511] Master 127.0.194.254:43979 requested a full tablet report, sending...
I20260926 03:43:57.841203 2526 ts_manager.cc:194] Registered new tserver with Master: 737f7fce56574ef6a528cd2a48e8056d (127.0.194.195:39703)
I20260926 03:43:57.842088 779 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu as pid 2876
I20260926 03:43:57.842100 2526 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.194.195@KRBTEST.COM'} at 127.0.194.195:39843
I20260926 03:43:57.842260 779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790394221564982-779-0/minicluster-data/ts-2/wal/instance
I20260926 03:43:57.842912 779 external_mini_cluster.cc:1428] Setting key c8df138fb2e915d1d54c767499cfe7a9
I20260926 03:43:57.846459 779 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 03:43:57 dist-test-slave-n2th krb5kdc[2481](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394237, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 03:43:57 dist-test-slave-n2th krb5kdc[2481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394237, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.194.254@KRBTEST.COM
I20260926 03:43:57.893164 2621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:57.880035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56044 (local address 127.0.194.254:43979)
0926 03:43:57.880195 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0926 03:43:57.880200 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:43:57.880532 (+ 332us) server_negotiation.cc:407] Connection header received
0926 03:43:57.880681 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:43:57.880684 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:43:57.880747 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:43:57.880840 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:43:57.882261 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:57.883103 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:57.884315 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:57.884706 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:57.884798 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:43:57.888403 (+ 3605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:43:57.888408 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:43:57.888411 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:43:57.888442 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:43:57.891061 (+ 2619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:43:57.891615 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:43:57.891619 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:43:57.891621 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:43:57.891730 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:43:57.892046 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:43:57.892048 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:43:57.892050 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:43:57.892296 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:43:57.892466 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0926 03:43:57.892701 (+ 235us) server_negotiation.cc:299] Negotiation successful
0926 03:43:57.892865 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 03:43:57.897119 2526 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56044:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 03:43:57.904030 2526 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 03:43:57.929077 3028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:57.919977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.193:36187 (local address 127.0.0.1:39770)
0926 03:43:57.922149 (+ 2172us) negotiation.cc:107] Waiting for socket to connect
0926 03:43:57.922167 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 03:43:57.922280 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:43:57.922888 (+ 608us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:43:57.922893 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:43:57.922919 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:43:57.923247 (+ 328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:43:57.923257 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:43:57.924936 (+ 1679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:43:57.924941 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:43:57.926608 (+ 1667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:43:57.926623 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:43:57.927988 (+ 1365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:43:57.927994 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:43:57.928187 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:43:57.928243 (+ 56us) client_negotiation.cc:769] Sending connection context
0926 03:43:57.928402 (+ 159us) client_negotiation.cc:240] Negotiation successful
0926 03:43:57.928591 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2033,"mutex_wait_us":49,"thread_start_us":1851,"threads_started":1}
I20260926 03:43:57.929853 3025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:57.916731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44504 (local address 127.0.194.194:42655)
0926 03:43:57.919823 (+ 3092us) server_negotiation.cc:206] Beginning negotiation
0926 03:43:57.919832 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:43:57.919863 (+ 31us) server_negotiation.cc:407] Connection header received
0926 03:43:57.919965 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:43:57.919973 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:43:57.920247 (+ 274us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:43:57.920428 (+ 181us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:43:57.922220 (+ 1792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:57.923593 (+ 1373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:57.926837 (+ 3244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:57.928263 (+ 1426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:57.928404 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:43:57.928606 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0926 03:43:57.929349 (+ 743us) server_negotiation.cc:299] Negotiation successful
0926 03:43:57.929578 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2904,"thread_start_us":2780,"threads_started":1}
I20260926 03:43:57.930143 3029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:57.920157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39770 (local address 127.0.194.193:36187)
0926 03:43:57.922112 (+ 1955us) server_negotiation.cc:206] Beginning negotiation
0926 03:43:57.922122 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 03:43:57.922278 (+ 156us) server_negotiation.cc:407] Connection header received
0926 03:43:57.922414 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:43:57.922423 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:43:57.922672 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:43:57.922898 (+ 226us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:43:57.923440 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:57.924766 (+ 1326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:57.926787 (+ 2021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:57.927794 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:57.927956 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:43:57.928185 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0926 03:43:57.929476 (+ 1291us) server_negotiation.cc:299] Negotiation successful
0926 03:43:57.929662 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1304,"thread_start_us":1178,"threads_started":1}
I20260926 03:43:57.929623 3026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:57.916574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.194:42655 (local address 127.0.0.1:44504)
0926 03:43:57.919001 (+ 2427us) negotiation.cc:107] Waiting for socket to connect
0926 03:43:57.919045 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 03:43:57.919221 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:43:57.920684 (+ 1463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:43:57.920694 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:43:57.920726 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:43:57.922017 (+ 1291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:43:57.922034 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:43:57.923714 (+ 1680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:43:57.923722 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:43:57.926600 (+ 2878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:43:57.926625 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:43:57.928806 (+ 2181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:43:57.928813 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:43:57.928986 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:43:57.929032 (+ 46us) client_negotiation.cc:769] Sending connection context
0926 03:43:57.929137 (+ 105us) client_negotiation.cc:240] Negotiation successful
0926 03:43:57.929280 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2176,"mutex_wait_us":186,"thread_start_us":2792,"threads_started":1}
I20260926 03:43:57.931231 3027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:57.917576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60094 (local address 127.0.194.195:39703)
0926 03:43:57.918381 (+ 805us) server_negotiation.cc:206] Beginning negotiation
0926 03:43:57.918394 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 03:43:57.919922 (+ 1528us) server_negotiation.cc:407] Connection header received
0926 03:43:57.920018 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:43:57.920024 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:43:57.920237 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:43:57.920428 (+ 191us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:43:57.922105 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:57.923625 (+ 1520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:57.926827 (+ 3202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:57.928263 (+ 1436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:57.928404 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:43:57.928627 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0926 03:43:57.930716 (+ 2089us) server_negotiation.cc:299] Negotiation successful
0926 03:43:57.930923 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":590,"thread_start_us":390,"threads_started":1}
I20260926 03:43:57.930586 3024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:57.917013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.195:39703 (local address 127.0.0.1:60094)
0926 03:43:57.919673 (+ 2660us) negotiation.cc:107] Waiting for socket to connect
0926 03:43:57.919692 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 03:43:57.919807 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:43:57.920723 (+ 916us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:43:57.920727 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:43:57.920742 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:43:57.921850 (+ 1108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:43:57.921870 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:43:57.923778 (+ 1908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:43:57.923782 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:43:57.926631 (+ 2849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:43:57.926639 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:43:57.929769 (+ 3130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:43:57.929776 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:43:57.929935 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:43:57.929980 (+ 45us) client_negotiation.cc:769] Sending connection context
0926 03:43:57.930064 (+ 84us) client_negotiation.cc:240] Negotiation successful
0926 03:43:57.930192 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2544,"mutex_wait_us":396,"thread_start_us":1572,"threads_started":1}
I20260926 03:43:57.935035 2802 tablet_service.cc:1467] Processing CreateTablet for tablet 12e1238edcbe4da891338ccef85ec9a5 (DEFAULT_TABLE table=test-table [id=7bfddf5e2b6644059aaf1f446942b200]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:43:57.935040 2663 tablet_service.cc:1467] Processing CreateTablet for tablet 12e1238edcbe4da891338ccef85ec9a5 (DEFAULT_TABLE table=test-table [id=7bfddf5e2b6644059aaf1f446942b200]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:43:57.936472 2663 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12e1238edcbe4da891338ccef85ec9a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:43:57.936499 2802 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12e1238edcbe4da891338ccef85ec9a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:43:57.937054 2941 tablet_service.cc:1467] Processing CreateTablet for tablet 12e1238edcbe4da891338ccef85ec9a5 (DEFAULT_TABLE table=test-table [id=7bfddf5e2b6644059aaf1f446942b200]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:43:57.938416 2941 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12e1238edcbe4da891338ccef85ec9a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:43:57.949036 3030 tablet_bootstrap.cc:492] T 12e1238edcbe4da891338ccef85ec9a5 P f20d775ac36948528aadf2fb37a3ed11: Bootstrap starting.
I20260926 03:43:57.952327 3030 tablet_bootstrap.cc:654] T 12e1238edcbe4da891338ccef85ec9a5 P f20d775ac36948528aadf2fb37a3ed11: Neither blocks nor log segments found. Creating new log.
I20260926 03:43:57.952857 3031 tablet_bootstrap.cc:492] T 12e1238edcbe4da891338ccef85ec9a5 P f496bcbb6d8644ad962eceeeac3f82ca: Bootstrap starting.
I20260926 03:43:57.954288 3030 log.cc:824] T 12e1238edcbe4da891338ccef85ec9a5 P f20d775ac36948528aadf2fb37a3ed11: Log is configured to *not* fsync() on all Append() calls
I20260926 03:43:57.955503 3032 tablet_bootstrap.cc:492] T 12e1238edcbe4da891338ccef85ec9a5 P 737f7fce56574ef6a528cd2a48e8056d: Bootstrap starting.
I20260926 03:43:57.956668 3031 tablet_bootstrap.cc:654] T 12e1238edcbe4da891338ccef85ec9a5 P f496bcbb6d8644ad962eceeeac3f82ca: Neither blocks nor log segments found. Creating new log.
I20260926 03:43:57.957837 3031 log.cc:824] T 12e1238edcbe4da891338ccef85ec9a5 P f496bcbb6d8644ad962eceeeac3f82ca: Log is configured to *not* fsync() on all Append() calls
I20260926 03:43:57.958781 3032 tablet_bootstrap.cc:654] T 12e1238edcbe4da891338ccef85ec9a5 P 737f7fce56574ef6a528cd2a48e8056d: Neither blocks nor log segments found. Creating new log.
I20260926 03:43:57.958972 3030 tablet_bootstrap.cc:492] T 12e1238edcbe4da891338ccef85ec9a5 P f20d775ac36948528aadf2fb37a3ed11: No bootstrap required, opened a new log
I20260926 03:43:57.959245 3030 ts_tablet_manager.cc:1397] T 12e1238edcbe4da891338ccef85ec9a5 P f20d775ac36948528aadf2fb37a3ed11: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20260926 03:43:57.959820 3032 log.cc:824] T 12e1238edcbe4da891338ccef85ec9a5 P 737f7fce56574ef6a528cd2a48e8056d: Log is configured to *not* fsync() on all Append() calls
I20260926 03:43:57.960881 3031 tablet_bootstrap.cc:492] T 12e1238edcbe4da891338ccef85ec9a5 P f496bcbb6d8644ad962eceeeac3f82ca: No bootstrap required, opened a new log
I20260926 03:43:57.961149 3031 ts_tablet_manager.cc:1397] T 12e1238edcbe4da891338ccef85ec9a5 P f496bcbb6d8644ad962eceeeac3f82ca: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260926 03:43:57.962803 3032 tablet_bootstrap.cc:492] T 12e1238edcbe4da891338ccef85ec9a5 P 737f7fce56574ef6a528cd2a48e8056d: No bootstrap required, opened a new log
I20260926 03:43:57.963061 3032 ts_tablet_manager.cc:1397] T 12e1238edcbe4da891338ccef85ec9a5 P 737f7fce56574ef6a528cd2a48e8056d: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.005s
I20260926 03:43:57.969838 3030 raft_consensus.cc:348] T 12e1238edcbe4da891338ccef85ec9a5 P f20d775ac36948528aadf2fb37a3ed11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20d775ac36948528aadf2fb37a3ed11" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 42655 } } peers { permanent_uuid: "f496bcbb6d8644ad962eceeeac3f82ca" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 36187 } } peers { permanent_uuid: "737f7fce56574ef6a528cd2a48e8056d" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 39703 } }
I20260926 03:43:57.970302 3030 raft_consensus.cc:374] T 12e1238edcbe4da891338ccef85ec9a5 P f20d775ac36948528aadf2fb37a3ed11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:43:57.970396 3030 raft_consensus.cc:729] T 12e1238edcbe4da891338ccef85ec9a5 P f20d775ac36948528aadf2fb37a3ed11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f20d775ac36948528aadf2fb37a3ed11, State: Initialized, Role: FOLLOWER
I20260926 03:43:57.970152 3032 raft_consensus.cc:348] T 12e1238edcbe4da891338ccef85ec9a5 P 737f7fce56574ef6a528cd2a48e8056d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20d775ac36948528aadf2fb37a3ed11" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 42655 } } peers { permanent_uuid: "f496bcbb6d8644ad962eceeeac3f82ca" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 36187 } } peers { permanent_uuid: "737f7fce56574ef6a528cd2a48e8056d" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 39703 } }
I20260926 03:43:57.970590 3032 raft_consensus.cc:374] T 12e1238edcbe4da891338ccef85ec9a5 P 737f7fce56574ef6a528cd2a48e8056d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:43:57.970716 3032 raft_consensus.cc:729] T 12e1238edcbe4da891338ccef85ec9a5 P 737f7fce56574ef6a528cd2a48e8056d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 737f7fce56574ef6a528cd2a48e8056d, State: Initialized, Role: FOLLOWER
I20260926 03:43:57.971063 3030 consensus_queue.cc:260] T 12e1238edcbe4da891338ccef85ec9a5 P f20d775ac36948528aadf2fb37a3ed11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20d775ac36948528aadf2fb37a3ed11" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 42655 } } peers { permanent_uuid: "f496bcbb6d8644ad962eceeeac3f82ca" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 36187 } } peers { permanent_uuid: "737f7fce56574ef6a528cd2a48e8056d" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 39703 } }
I20260926 03:43:57.971326 3032 consensus_queue.cc:260] T 12e1238edcbe4da891338ccef85ec9a5 P 737f7fce56574ef6a528cd2a48e8056d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20d775ac36948528aadf2fb37a3ed11" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 42655 } } peers { permanent_uuid: "f496bcbb6d8644ad962eceeeac3f82ca" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 36187 } } peers { permanent_uuid: "737f7fce56574ef6a528cd2a48e8056d" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 39703 } }
I20260926 03:43:57.971557 3031 raft_consensus.cc:348] T 12e1238edcbe4da891338ccef85ec9a5 P f496bcbb6d8644ad962eceeeac3f82ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20d775ac36948528aadf2fb37a3ed11" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 42655 } } peers { permanent_uuid: "f496bcbb6d8644ad962eceeeac3f82ca" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 36187 } } peers { permanent_uuid: "737f7fce56574ef6a528cd2a48e8056d" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 39703 } }
I20260926 03:43:57.971990 3031 raft_consensus.cc:374] T 12e1238edcbe4da891338ccef85ec9a5 P f496bcbb6d8644ad962eceeeac3f82ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:43:57.972088 3031 raft_consensus.cc:729] T 12e1238edcbe4da891338ccef85ec9a5 P f496bcbb6d8644ad962eceeeac3f82ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f496bcbb6d8644ad962eceeeac3f82ca, State: Initialized, Role: FOLLOWER
I20260926 03:43:57.972431 2872 heartbeater.cc:503] Master 127.0.194.254:43979 was elected leader, sending a full tablet report...
I20260926 03:43:57.972620 3031 consensus_queue.cc:260] T 12e1238edcbe4da891338ccef85ec9a5 P f496bcbb6d8644ad962eceeeac3f82ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20d775ac36948528aadf2fb37a3ed11" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 42655 } } peers { permanent_uuid: "f496bcbb6d8644ad962eceeeac3f82ca" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 36187 } } peers { permanent_uuid: "737f7fce56574ef6a528cd2a48e8056d" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 39703 } }
I20260926 03:43:57.972988 3030 ts_tablet_manager.cc:1428] T 12e1238edcbe4da891338ccef85ec9a5 P f20d775ac36948528aadf2fb37a3ed11: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260926 03:43:57.973839 3011 heartbeater.cc:503] Master 127.0.194.254:43979 was elected leader, sending a full tablet report...
I20260926 03:43:57.974426 3031 ts_tablet_manager.cc:1428] T 12e1238edcbe4da891338ccef85ec9a5 P f496bcbb6d8644ad962eceeeac3f82ca: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260926 03:43:57.975119 3032 ts_tablet_manager.cc:1428] T 12e1238edcbe4da891338ccef85ec9a5 P 737f7fce56574ef6a528cd2a48e8056d: Time spent starting tablet: real 0.012s user 0.007s sys 0.003s
I20260926 03:43:57.991317 3036 raft_consensus.cc:482] T 12e1238edcbe4da891338ccef85ec9a5 P f20d775ac36948528aadf2fb37a3ed11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 03:43:57.991572 3036 raft_consensus.cc:504] T 12e1238edcbe4da891338ccef85ec9a5 P f20d775ac36948528aadf2fb37a3ed11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20d775ac36948528aadf2fb37a3ed11" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 42655 } } peers { permanent_uuid: "f496bcbb6d8644ad962eceeeac3f82ca" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 36187 } } peers { permanent_uuid: "737f7fce56574ef6a528cd2a48e8056d" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 39703 } }
I20260926 03:43:57.993106 3036 leader_election.cc:288] T 12e1238edcbe4da891338ccef85ec9a5 P f20d775ac36948528aadf2fb37a3ed11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f496bcbb6d8644ad962eceeeac3f82ca (127.0.194.193:36187), 737f7fce56574ef6a528cd2a48e8056d (127.0.194.195:39703)
I20260926 03:43:58.000626 3027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:57.993849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.194:56409 (local address 127.0.194.195:39703)
0926 03:43:57.994028 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0926 03:43:57.994036 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:43:57.995246 (+ 1210us) server_negotiation.cc:407] Connection header received
0926 03:43:57.995529 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:43:57.995534 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:43:57.995595 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:43:57.995673 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:43:57.996360 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:57.997058 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:57.998941 (+ 1883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:57.999693 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:57.999889 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:43:57.999967 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 03:43:58.000245 (+ 278us) server_negotiation.cc:299] Negotiation successful
0926 03:43:58.000368 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 03:43:58.000792 3039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:57.993364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.195:39703 (local address 127.0.194.194:56409)
0926 03:43:57.995133 (+ 1769us) negotiation.cc:107] Waiting for socket to connect
0926 03:43:57.995153 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 03:43:57.995321 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:43:57.995772 (+ 451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:43:57.995778 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:43:57.995806 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:43:57.996207 (+ 401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:43:57.996226 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:43:57.997192 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:43:57.997199 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:43:57.998786 (+ 1587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:43:57.998808 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:43:57.999917 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:43:57.999923 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:43:58.000086 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:43:58.000114 (+ 28us) client_negotiation.cc:769] Sending connection context
0926 03:43:58.000220 (+ 106us) client_negotiation.cc:240] Negotiation successful
0926 03:43:58.000353 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1650,"mutex_wait_us":100,"thread_start_us":1486,"threads_started":1}
I20260926 03:43:58.000801 3029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:57.993651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.194:37595 (local address 127.0.194.193:36187)
0926 03:43:57.993850 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0926 03:43:57.993856 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:43:57.995699 (+ 1843us) server_negotiation.cc:407] Connection header received
0926 03:43:57.995884 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:43:57.995887 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:43:57.995945 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:43:57.996015 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:43:57.996671 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:57.997368 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:57.999122 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:57.999908 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:58.000098 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:43:58.000169 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 03:43:58.000436 (+ 267us) server_negotiation.cc:299] Negotiation successful
0926 03:43:58.000563 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 03:43:58.000967 3040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:57.993489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.193:36187 (local address 127.0.194.194:37595)
0926 03:43:57.995595 (+ 2106us) negotiation.cc:107] Waiting for socket to connect
0926 03:43:57.995608 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 03:43:57.995747 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:43:57.996065 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:43:57.996070 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:43:57.996095 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:43:57.996518 (+ 423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:43:57.996526 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:43:57.997510 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:43:57.997517 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:43:57.998977 (+ 1460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:43:57.998990 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:43:58.000122 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:43:58.000128 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:43:58.000284 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:43:58.000306 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 03:43:58.000410 (+ 104us) client_negotiation.cc:240] Negotiation successful
0926 03:43:58.000535 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2019,"mutex_wait_us":5,"thread_start_us":1821,"threads_started":1}
I20260926 03:43:58.001896 2687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12e1238edcbe4da891338ccef85ec9a5" candidate_uuid: "f20d775ac36948528aadf2fb37a3ed11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f496bcbb6d8644ad962eceeeac3f82ca" is_pre_election: true
I20260926 03:43:58.001895 2961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12e1238edcbe4da891338ccef85ec9a5" candidate_uuid: "f20d775ac36948528aadf2fb37a3ed11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "737f7fce56574ef6a528cd2a48e8056d" is_pre_election: true
I20260926 03:43:58.002466 2961 raft_consensus.cc:2454] T 12e1238edcbe4da891338ccef85ec9a5 P 737f7fce56574ef6a528cd2a48e8056d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f20d775ac36948528aadf2fb37a3ed11 in term 0.
I20260926 03:43:58.002466 2687 raft_consensus.cc:2454] T 12e1238edcbe4da891338ccef85ec9a5 P f496bcbb6d8644ad962eceeeac3f82ca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f20d775ac36948528aadf2fb37a3ed11 in term 0.
I20260926 03:43:58.003239 2753 leader_election.cc:302] T 12e1238edcbe4da891338ccef85ec9a5 P f20d775ac36948528aadf2fb37a3ed11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 737f7fce56574ef6a528cd2a48e8056d, f20d775ac36948528aadf2fb37a3ed11; no voters:
I20260926 03:43:58.003728 3036 raft_consensus.cc:2781] T 12e1238edcbe4da891338ccef85ec9a5 P f20d775ac36948528aadf2fb37a3ed11 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 03:43:58.003883 3036 raft_consensus.cc:482] T 12e1238edcbe4da891338ccef85ec9a5 P f20d775ac36948528aadf2fb37a3ed11 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 03:43:58.003988 3036 raft_consensus.cc:3037] T 12e1238edcbe4da891338ccef85ec9a5 P f20d775ac36948528aadf2fb37a3ed11 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:43:58.006382 3036 raft_consensus.cc:504] T 12e1238edcbe4da891338ccef85ec9a5 P f20d775ac36948528aadf2fb37a3ed11 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20d775ac36948528aadf2fb37a3ed11" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 42655 } } peers { permanent_uuid: "f496bcbb6d8644ad962eceeeac3f82ca" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 36187 } } peers { permanent_uuid: "737f7fce56574ef6a528cd2a48e8056d" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 39703 } }
I20260926 03:43:58.007072 3036 leader_election.cc:288] T 12e1238edcbe4da891338ccef85ec9a5 P f20d775ac36948528aadf2fb37a3ed11 [CANDIDATE]: Term 1 election: Requested vote from peers f496bcbb6d8644ad962eceeeac3f82ca (127.0.194.193:36187), 737f7fce56574ef6a528cd2a48e8056d (127.0.194.195:39703)
I20260926 03:43:58.007525 2687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12e1238edcbe4da891338ccef85ec9a5" candidate_uuid: "f20d775ac36948528aadf2fb37a3ed11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f496bcbb6d8644ad962eceeeac3f82ca"
I20260926 03:43:58.007583 2961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12e1238edcbe4da891338ccef85ec9a5" candidate_uuid: "f20d775ac36948528aadf2fb37a3ed11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "737f7fce56574ef6a528cd2a48e8056d"
I20260926 03:43:58.007723 2687 raft_consensus.cc:3037] T 12e1238edcbe4da891338ccef85ec9a5 P f496bcbb6d8644ad962eceeeac3f82ca [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:43:58.007843 2961 raft_consensus.cc:3037] T 12e1238edcbe4da891338ccef85ec9a5 P 737f7fce56574ef6a528cd2a48e8056d [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:43:58.009976 2687 raft_consensus.cc:2454] T 12e1238edcbe4da891338ccef85ec9a5 P f496bcbb6d8644ad962eceeeac3f82ca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f20d775ac36948528aadf2fb37a3ed11 in term 1.
I20260926 03:43:58.010061 2961 raft_consensus.cc:2454] T 12e1238edcbe4da891338ccef85ec9a5 P 737f7fce56574ef6a528cd2a48e8056d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f20d775ac36948528aadf2fb37a3ed11 in term 1.
I20260926 03:43:58.010440 2756 leader_election.cc:302] T 12e1238edcbe4da891338ccef85ec9a5 P f20d775ac36948528aadf2fb37a3ed11 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f20d775ac36948528aadf2fb37a3ed11, f496bcbb6d8644ad962eceeeac3f82ca; no voters:
I20260926 03:43:58.010785 3036 raft_consensus.cc:2781] T 12e1238edcbe4da891338ccef85ec9a5 P f20d775ac36948528aadf2fb37a3ed11 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:43:58.011281 3036 raft_consensus.cc:686] T 12e1238edcbe4da891338ccef85ec9a5 P f20d775ac36948528aadf2fb37a3ed11 [term 1 LEADER]: Becoming Leader. State: Replica: f20d775ac36948528aadf2fb37a3ed11, State: Running, Role: LEADER
I20260926 03:43:58.011842 3036 consensus_queue.cc:237] T 12e1238edcbe4da891338ccef85ec9a5 P f20d775ac36948528aadf2fb37a3ed11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20d775ac36948528aadf2fb37a3ed11" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 42655 } } peers { permanent_uuid: "f496bcbb6d8644ad962eceeeac3f82ca" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 36187 } } peers { permanent_uuid: "737f7fce56574ef6a528cd2a48e8056d" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 39703 } }
I20260926 03:43:58.016211 2526 catalog_manager.cc:5495] T 12e1238edcbe4da891338ccef85ec9a5 P f20d775ac36948528aadf2fb37a3ed11 reported cstate change: term changed from 0 to 1, leader changed from <none> to f20d775ac36948528aadf2fb37a3ed11 (127.0.194.194). New cstate: current_term: 1 leader_uuid: "f20d775ac36948528aadf2fb37a3ed11" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20d775ac36948528aadf2fb37a3ed11" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 42655 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f496bcbb6d8644ad962eceeeac3f82ca" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 36187 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "737f7fce56574ef6a528cd2a48e8056d" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 39703 } health_report { overall_health: UNKNOWN } } }
W20260926 03:43:58.043660 2734 tablet.cc:2369] T 12e1238edcbe4da891338ccef85ec9a5 P f496bcbb6d8644ad962eceeeac3f82ca: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 26 03:43:58 dist-test-slave-n2th krb5kdc[2481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394236, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.254@KRBTEST.COM for kudu/127.0.194.254@KRBTEST.COM
I20260926 03:43:58.059113 3048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:58.045732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:43979 (local address 127.0.0.1:56058)
0926 03:43:58.046340 (+ 608us) negotiation.cc:107] Waiting for socket to connect
0926 03:43:58.046356 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 03:43:58.046539 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:43:58.046932 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:43:58.046936 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:43:58.047647 (+ 711us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:43:58.047912 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:43:58.047920 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:43:58.048913 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:43:58.048921 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:43:58.049879 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:43:58.049889 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:43:58.050684 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:43:58.050690 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:43:58.050881 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:43:58.051524 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:43:58.051563 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:43:58.053807 (+ 2244us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:43:58.056697 (+ 2890us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:43:58.056709 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:43:58.056715 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:43:58.057169 (+ 454us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:43:58.057499 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:43:58.057502 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:43:58.057511 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:43:58.057642 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:43:58.058156 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:43:58.058162 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:43:58.058317 (+ 155us) client_negotiation.cc:769] Sending connection context
0926 03:43:58.058460 (+ 143us) client_negotiation.cc:240] Negotiation successful
0926 03:43:58.058674 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":344,"threads_started":1}
I20260926 03:43:58.059118 2621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:58.045665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56058 (local address 127.0.194.254:43979)
0926 03:43:58.045843 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0926 03:43:58.045848 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:43:58.046452 (+ 604us) server_negotiation.cc:407] Connection header received
0926 03:43:58.046669 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:43:58.046679 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:43:58.046791 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:43:58.046911 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:43:58.048074 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:58.048787 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:58.050055 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:58.050482 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:58.050643 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:43:58.053998 (+ 3355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:43:58.054003 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:43:58.054006 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:43:58.054044 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:43:58.056549 (+ 2505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:43:58.057275 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:43:58.057278 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:43:58.057281 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:43:58.057399 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:43:58.057739 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:43:58.057742 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:43:58.057744 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:43:58.058025 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:43:58.058174 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0926 03:43:58.058545 (+ 371us) server_negotiation.cc:299] Negotiation successful
0926 03:43:58.058861 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":77,"server-negotiator.queue_time_us":59}
W20260926 03:43:58.061775 2873 tablet.cc:2369] T 12e1238edcbe4da891338ccef85ec9a5 P f20d775ac36948528aadf2fb37a3ed11: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 03:43:58.064512 2526 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.194.254@KRBTEST.COM'} at 127.0.0.1:56058:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260926 03:43:58.065482 2526 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 03:43:58.072929 2941 tablet_service.cc:1467] Processing CreateTablet for tablet 3b8a073ce4d5458abdc874e9acd190b5 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f8c78253d3244910bdc9f8e0a991e87f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 03:43:58.072723 2802 tablet_service.cc:1467] Processing CreateTablet for tablet 3b8a073ce4d5458abdc874e9acd190b5 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f8c78253d3244910bdc9f8e0a991e87f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 03:43:58.072723 2663 tablet_service.cc:1467] Processing CreateTablet for tablet 3b8a073ce4d5458abdc874e9acd190b5 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f8c78253d3244910bdc9f8e0a991e87f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 03:43:58.073791 2941 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b8a073ce4d5458abdc874e9acd190b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:43:58.073794 2663 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b8a073ce4d5458abdc874e9acd190b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:43:58.073794 2802 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b8a073ce4d5458abdc874e9acd190b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:43:58.080153 3032 tablet_bootstrap.cc:492] T 3b8a073ce4d5458abdc874e9acd190b5 P 737f7fce56574ef6a528cd2a48e8056d: Bootstrap starting.
I20260926 03:43:58.081200 3030 tablet_bootstrap.cc:492] T 3b8a073ce4d5458abdc874e9acd190b5 P f20d775ac36948528aadf2fb37a3ed11: Bootstrap starting.
I20260926 03:43:58.081200 3031 tablet_bootstrap.cc:492] T 3b8a073ce4d5458abdc874e9acd190b5 P f496bcbb6d8644ad962eceeeac3f82ca: Bootstrap starting.
I20260926 03:43:58.081959 3032 tablet_bootstrap.cc:654] T 3b8a073ce4d5458abdc874e9acd190b5 P 737f7fce56574ef6a528cd2a48e8056d: Neither blocks nor log segments found. Creating new log.
I20260926 03:43:58.082880 3031 tablet_bootstrap.cc:654] T 3b8a073ce4d5458abdc874e9acd190b5 P f496bcbb6d8644ad962eceeeac3f82ca: Neither blocks nor log segments found. Creating new log.
I20260926 03:43:58.082890 3030 tablet_bootstrap.cc:654] T 3b8a073ce4d5458abdc874e9acd190b5 P f20d775ac36948528aadf2fb37a3ed11: Neither blocks nor log segments found. Creating new log.
I20260926 03:43:58.084625 3032 tablet_bootstrap.cc:492] T 3b8a073ce4d5458abdc874e9acd190b5 P 737f7fce56574ef6a528cd2a48e8056d: No bootstrap required, opened a new log
I20260926 03:43:58.084772 3032 ts_tablet_manager.cc:1397] T 3b8a073ce4d5458abdc874e9acd190b5 P 737f7fce56574ef6a528cd2a48e8056d: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.001s
I20260926 03:43:58.085639 3032 raft_consensus.cc:348] T 3b8a073ce4d5458abdc874e9acd190b5 P 737f7fce56574ef6a528cd2a48e8056d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f496bcbb6d8644ad962eceeeac3f82ca" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 36187 } } peers { permanent_uuid: "f20d775ac36948528aadf2fb37a3ed11" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 42655 } } peers { permanent_uuid: "737f7fce56574ef6a528cd2a48e8056d" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 39703 } }
I20260926 03:43:58.085923 3032 raft_consensus.cc:374] T 3b8a073ce4d5458abdc874e9acd190b5 P 737f7fce56574ef6a528cd2a48e8056d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:43:58.086028 3032 raft_consensus.cc:729] T 3b8a073ce4d5458abdc874e9acd190b5 P 737f7fce56574ef6a528cd2a48e8056d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 737f7fce56574ef6a528cd2a48e8056d, State: Initialized, Role: FOLLOWER
I20260926 03:43:58.086346 3032 consensus_queue.cc:260] T 3b8a073ce4d5458abdc874e9acd190b5 P 737f7fce56574ef6a528cd2a48e8056d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f496bcbb6d8644ad962eceeeac3f82ca" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 36187 } } peers { permanent_uuid: "f20d775ac36948528aadf2fb37a3ed11" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 42655 } } peers { permanent_uuid: "737f7fce56574ef6a528cd2a48e8056d" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 39703 } }
I20260926 03:43:58.087040 3032 ts_tablet_manager.cc:1428] T 3b8a073ce4d5458abdc874e9acd190b5 P 737f7fce56574ef6a528cd2a48e8056d: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20260926 03:43:58.086923 3037 tablet_replica.cc:439] T 3b8a073ce4d5458abdc874e9acd190b5 P 737f7fce56574ef6a528cd2a48e8056d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f496bcbb6d8644ad962eceeeac3f82ca" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 36187 } } peers { permanent_uuid: "f20d775ac36948528aadf2fb37a3ed11" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 42655 } } peers { permanent_uuid: "737f7fce56574ef6a528cd2a48e8056d" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 39703 } } }
I20260926 03:43:58.087286 3037 tablet_replica.cc:442] T 3b8a073ce4d5458abdc874e9acd190b5 P 737f7fce56574ef6a528cd2a48e8056d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 03:43:58.087486 3031 tablet_bootstrap.cc:492] T 3b8a073ce4d5458abdc874e9acd190b5 P f496bcbb6d8644ad962eceeeac3f82ca: No bootstrap required, opened a new log
I20260926 03:43:58.087639 3031 ts_tablet_manager.cc:1397] T 3b8a073ce4d5458abdc874e9acd190b5 P f496bcbb6d8644ad962eceeeac3f82ca: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260926 03:43:58.087666 3030 tablet_bootstrap.cc:492] T 3b8a073ce4d5458abdc874e9acd190b5 P f20d775ac36948528aadf2fb37a3ed11: No bootstrap required, opened a new log
I20260926 03:43:58.088048 3030 ts_tablet_manager.cc:1397] T 3b8a073ce4d5458abdc874e9acd190b5 P f20d775ac36948528aadf2fb37a3ed11: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
W20260926 03:43:58.088328 3012 tablet.cc:2369] T 12e1238edcbe4da891338ccef85ec9a5 P 737f7fce56574ef6a528cd2a48e8056d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 03:43:58.088426 3031 raft_consensus.cc:348] T 3b8a073ce4d5458abdc874e9acd190b5 P f496bcbb6d8644ad962eceeeac3f82ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f496bcbb6d8644ad962eceeeac3f82ca" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 36187 } } peers { permanent_uuid: "f20d775ac36948528aadf2fb37a3ed11" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 42655 } } peers { permanent_uuid: "737f7fce56574ef6a528cd2a48e8056d" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 39703 } }
I20260926 03:43:58.088687 3031 raft_consensus.cc:374] T 3b8a073ce4d5458abdc874e9acd190b5 P f496bcbb6d8644ad962eceeeac3f82ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:43:58.088771 3031 raft_consensus.cc:729] T 3b8a073ce4d5458abdc874e9acd190b5 P f496bcbb6d8644ad962eceeeac3f82ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f496bcbb6d8644ad962eceeeac3f82ca, State: Initialized, Role: FOLLOWER
I20260926 03:43:58.088861 3030 raft_consensus.cc:348] T 3b8a073ce4d5458abdc874e9acd190b5 P f20d775ac36948528aadf2fb37a3ed11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f496bcbb6d8644ad962eceeeac3f82ca" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 36187 } } peers { permanent_uuid: "f20d775ac36948528aadf2fb37a3ed11" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 42655 } } peers { permanent_uuid: "737f7fce56574ef6a528cd2a48e8056d" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 39703 } }
I20260926 03:43:58.089107 3030 raft_consensus.cc:374] T 3b8a073ce4d5458abdc874e9acd190b5 P f20d775ac36948528aadf2fb37a3ed11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:43:58.089198 3030 raft_consensus.cc:729] T 3b8a073ce4d5458abdc874e9acd190b5 P f20d775ac36948528aadf2fb37a3ed11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f20d775ac36948528aadf2fb37a3ed11, State: Initialized, Role: FOLLOWER
I20260926 03:43:58.089429 3030 consensus_queue.cc:260] T 3b8a073ce4d5458abdc874e9acd190b5 P f20d775ac36948528aadf2fb37a3ed11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f496bcbb6d8644ad962eceeeac3f82ca" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 36187 } } peers { permanent_uuid: "f20d775ac36948528aadf2fb37a3ed11" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 42655 } } peers { permanent_uuid: "737f7fce56574ef6a528cd2a48e8056d" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 39703 } }
I20260926 03:43:58.089061 3031 consensus_queue.cc:260] T 3b8a073ce4d5458abdc874e9acd190b5 P f496bcbb6d8644ad962eceeeac3f82ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f496bcbb6d8644ad962eceeeac3f82ca" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 36187 } } peers { permanent_uuid: "f20d775ac36948528aadf2fb37a3ed11" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 42655 } } peers { permanent_uuid: "737f7fce56574ef6a528cd2a48e8056d" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 39703 } }
I20260926 03:43:58.090050 3030 ts_tablet_manager.cc:1428] T 3b8a073ce4d5458abdc874e9acd190b5 P f20d775ac36948528aadf2fb37a3ed11: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260926 03:43:58.089884 3036 tablet_replica.cc:439] T 3b8a073ce4d5458abdc874e9acd190b5 P f20d775ac36948528aadf2fb37a3ed11: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f496bcbb6d8644ad962eceeeac3f82ca" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 36187 } } peers { permanent_uuid: "f20d775ac36948528aadf2fb37a3ed11" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 42655 } } peers { permanent_uuid: "737f7fce56574ef6a528cd2a48e8056d" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 39703 } } }
I20260926 03:43:58.090191 3036 tablet_replica.cc:442] T 3b8a073ce4d5458abdc874e9acd190b5 P f20d775ac36948528aadf2fb37a3ed11: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 03:43:58.090054 3038 tablet_replica.cc:439] T 3b8a073ce4d5458abdc874e9acd190b5 P f496bcbb6d8644ad962eceeeac3f82ca: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f496bcbb6d8644ad962eceeeac3f82ca" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 36187 } } peers { permanent_uuid: "f20d775ac36948528aadf2fb37a3ed11" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 42655 } } peers { permanent_uuid: "737f7fce56574ef6a528cd2a48e8056d" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 39703 } } }
I20260926 03:43:58.090379 3038 tablet_replica.cc:442] T 3b8a073ce4d5458abdc874e9acd190b5 P f496bcbb6d8644ad962eceeeac3f82ca: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 03:43:58.090844 3031 ts_tablet_manager.cc:1428] T 3b8a073ce4d5458abdc874e9acd190b5 P f496bcbb6d8644ad962eceeeac3f82ca: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260926 03:43:58.326165 3038 raft_consensus.cc:482] T 3b8a073ce4d5458abdc874e9acd190b5 P f496bcbb6d8644ad962eceeeac3f82ca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 03:43:58.326428 3038 raft_consensus.cc:504] T 3b8a073ce4d5458abdc874e9acd190b5 P f496bcbb6d8644ad962eceeeac3f82ca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f496bcbb6d8644ad962eceeeac3f82ca" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 36187 } } peers { permanent_uuid: "f20d775ac36948528aadf2fb37a3ed11" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 42655 } } peers { permanent_uuid: "737f7fce56574ef6a528cd2a48e8056d" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 39703 } }
I20260926 03:43:58.328004 3038 leader_election.cc:288] T 3b8a073ce4d5458abdc874e9acd190b5 P f496bcbb6d8644ad962eceeeac3f82ca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f20d775ac36948528aadf2fb37a3ed11 (127.0.194.194:42655), 737f7fce56574ef6a528cd2a48e8056d (127.0.194.195:39703)
I20260926 03:43:58.335625 3027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:58.328573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.193:58497 (local address 127.0.194.195:39703)
0926 03:43:58.328749 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0926 03:43:58.328755 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:43:58.329437 (+ 682us) server_negotiation.cc:407] Connection header received
0926 03:43:58.329624 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:43:58.329631 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:43:58.329710 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:43:58.329832 (+ 122us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:43:58.330755 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:58.331908 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:58.333832 (+ 1924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:58.334569 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:58.334771 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:43:58.334854 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 03:43:58.335300 (+ 446us) server_negotiation.cc:299] Negotiation successful
0926 03:43:58.335440 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 03:43:58.335675 3025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:58.328562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.193:39613 (local address 127.0.194.194:42655)
0926 03:43:58.328749 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0926 03:43:58.328756 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:43:58.329438 (+ 682us) server_negotiation.cc:407] Connection header received
0926 03:43:58.329624 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:43:58.329631 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:43:58.329710 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:43:58.329818 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:43:58.330750 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:58.331908 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:58.333906 (+ 1998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:58.334674 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:58.334912 (+ 238us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:43:58.334984 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 03:43:58.335267 (+ 283us) server_negotiation.cc:299] Negotiation successful
0926 03:43:58.335404 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 03:43:58.335789 3053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:58.328361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.194:42655 (local address 127.0.194.193:39613)
0926 03:43:58.329323 (+ 962us) negotiation.cc:107] Waiting for socket to connect
0926 03:43:58.329339 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 03:43:58.329475 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:43:58.329842 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:43:58.329847 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:43:58.329909 (+ 62us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:43:58.330514 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:43:58.330527 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:43:58.332071 (+ 1544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:43:58.332076 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:43:58.333748 (+ 1672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:43:58.333761 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:43:58.334933 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:43:58.334939 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:43:58.335107 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:43:58.335130 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 03:43:58.335232 (+ 102us) client_negotiation.cc:240] Negotiation successful
0926 03:43:58.335375 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":846,"mutex_wait_us":80,"spinlock_wait_cycles":17024,"thread_start_us":602,"threads_started":1}
I20260926 03:43:58.336357 3052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:58.328334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.195:39703 (local address 127.0.194.193:58497)
0926 03:43:58.329275 (+ 941us) negotiation.cc:107] Waiting for socket to connect
0926 03:43:58.329297 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 03:43:58.329481 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:43:58.329842 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:43:58.329847 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:43:58.329912 (+ 65us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:43:58.330513 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:43:58.330527 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:43:58.332119 (+ 1592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:43:58.332123 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:43:58.333667 (+ 1544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:43:58.333682 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:43:58.334819 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:43:58.334826 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:43:58.335044 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:43:58.335129 (+ 85us) client_negotiation.cc:769] Sending connection context
0926 03:43:58.335870 (+ 741us) client_negotiation.cc:240] Negotiation successful
0926 03:43:58.336003 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":806,"mutex_wait_us":71,"spinlock_wait_cycles":90240,"thread_start_us":601,"threads_started":1}
I20260926 03:43:58.336745 2827 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b8a073ce4d5458abdc874e9acd190b5" candidate_uuid: "f496bcbb6d8644ad962eceeeac3f82ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f20d775ac36948528aadf2fb37a3ed11" is_pre_election: true
I20260926 03:43:58.337007 2961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b8a073ce4d5458abdc874e9acd190b5" candidate_uuid: "f496bcbb6d8644ad962eceeeac3f82ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "737f7fce56574ef6a528cd2a48e8056d" is_pre_election: true
I20260926 03:43:58.337167 2827 raft_consensus.cc:2454] T 3b8a073ce4d5458abdc874e9acd190b5 P f20d775ac36948528aadf2fb37a3ed11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f496bcbb6d8644ad962eceeeac3f82ca in term 0.
I20260926 03:43:58.337266 2961 raft_consensus.cc:2454] T 3b8a073ce4d5458abdc874e9acd190b5 P 737f7fce56574ef6a528cd2a48e8056d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f496bcbb6d8644ad962eceeeac3f82ca in term 0.
I20260926 03:43:58.337771 2613 leader_election.cc:302] T 3b8a073ce4d5458abdc874e9acd190b5 P f496bcbb6d8644ad962eceeeac3f82ca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 737f7fce56574ef6a528cd2a48e8056d, f20d775ac36948528aadf2fb37a3ed11, f496bcbb6d8644ad962eceeeac3f82ca; no voters:
I20260926 03:43:58.338261 3038 raft_consensus.cc:2781] T 3b8a073ce4d5458abdc874e9acd190b5 P f496bcbb6d8644ad962eceeeac3f82ca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 03:43:58.338409 3038 raft_consensus.cc:482] T 3b8a073ce4d5458abdc874e9acd190b5 P f496bcbb6d8644ad962eceeeac3f82ca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 03:43:58.338521 3038 raft_consensus.cc:3037] T 3b8a073ce4d5458abdc874e9acd190b5 P f496bcbb6d8644ad962eceeeac3f82ca [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:43:58.340829 3038 raft_consensus.cc:504] T 3b8a073ce4d5458abdc874e9acd190b5 P f496bcbb6d8644ad962eceeeac3f82ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f496bcbb6d8644ad962eceeeac3f82ca" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 36187 } } peers { permanent_uuid: "f20d775ac36948528aadf2fb37a3ed11" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 42655 } } peers { permanent_uuid: "737f7fce56574ef6a528cd2a48e8056d" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 39703 } }
I20260926 03:43:58.341419 3038 leader_election.cc:288] T 3b8a073ce4d5458abdc874e9acd190b5 P f496bcbb6d8644ad962eceeeac3f82ca [CANDIDATE]: Term 1 election: Requested vote from peers f20d775ac36948528aadf2fb37a3ed11 (127.0.194.194:42655), 737f7fce56574ef6a528cd2a48e8056d (127.0.194.195:39703)
I20260926 03:43:58.341908 2961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b8a073ce4d5458abdc874e9acd190b5" candidate_uuid: "f496bcbb6d8644ad962eceeeac3f82ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "737f7fce56574ef6a528cd2a48e8056d"
I20260926 03:43:58.341925 2827 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b8a073ce4d5458abdc874e9acd190b5" candidate_uuid: "f496bcbb6d8644ad962eceeeac3f82ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f20d775ac36948528aadf2fb37a3ed11"
I20260926 03:43:58.342137 2961 raft_consensus.cc:3037] T 3b8a073ce4d5458abdc874e9acd190b5 P 737f7fce56574ef6a528cd2a48e8056d [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:43:58.342128 2827 raft_consensus.cc:3037] T 3b8a073ce4d5458abdc874e9acd190b5 P f20d775ac36948528aadf2fb37a3ed11 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:43:58.344368 2961 raft_consensus.cc:2454] T 3b8a073ce4d5458abdc874e9acd190b5 P 737f7fce56574ef6a528cd2a48e8056d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f496bcbb6d8644ad962eceeeac3f82ca in term 1.
I20260926 03:43:58.344422 2827 raft_consensus.cc:2454] T 3b8a073ce4d5458abdc874e9acd190b5 P f20d775ac36948528aadf2fb37a3ed11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f496bcbb6d8644ad962eceeeac3f82ca in term 1.
I20260926 03:43:58.344862 2613 leader_election.cc:302] T 3b8a073ce4d5458abdc874e9acd190b5 P f496bcbb6d8644ad962eceeeac3f82ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 737f7fce56574ef6a528cd2a48e8056d, f496bcbb6d8644ad962eceeeac3f82ca; no voters:
I20260926 03:43:58.345191 3038 raft_consensus.cc:2781] T 3b8a073ce4d5458abdc874e9acd190b5 P f496bcbb6d8644ad962eceeeac3f82ca [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:43:58.345649 3038 raft_consensus.cc:686] T 3b8a073ce4d5458abdc874e9acd190b5 P f496bcbb6d8644ad962eceeeac3f82ca [term 1 LEADER]: Becoming Leader. State: Replica: f496bcbb6d8644ad962eceeeac3f82ca, State: Running, Role: LEADER
I20260926 03:43:58.346143 3038 consensus_queue.cc:237] T 3b8a073ce4d5458abdc874e9acd190b5 P f496bcbb6d8644ad962eceeeac3f82ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f496bcbb6d8644ad962eceeeac3f82ca" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 36187 } } peers { permanent_uuid: "f20d775ac36948528aadf2fb37a3ed11" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 42655 } } peers { permanent_uuid: "737f7fce56574ef6a528cd2a48e8056d" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 39703 } }
I20260926 03:43:58.349467 3054 tablet_replica.cc:439] T 3b8a073ce4d5458abdc874e9acd190b5 P f496bcbb6d8644ad962eceeeac3f82ca: TxnStatusTablet state changed. Reason: New leader f496bcbb6d8644ad962eceeeac3f82ca. Latest consensus state: current_term: 1 leader_uuid: "f496bcbb6d8644ad962eceeeac3f82ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f496bcbb6d8644ad962eceeeac3f82ca" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 36187 } } peers { permanent_uuid: "f20d775ac36948528aadf2fb37a3ed11" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 42655 } } peers { permanent_uuid: "737f7fce56574ef6a528cd2a48e8056d" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 39703 } } }
I20260926 03:43:58.350116 3054 tablet_replica.cc:442] T 3b8a073ce4d5458abdc874e9acd190b5 P f496bcbb6d8644ad962eceeeac3f82ca: This TxnStatusTablet replica's current role is: LEADER
I20260926 03:43:58.350855 3056 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260926 03:43:58.351145 3056 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260926 03:43:58.350953 2524 catalog_manager.cc:5495] T 3b8a073ce4d5458abdc874e9acd190b5 P f496bcbb6d8644ad962eceeeac3f82ca reported cstate change: term changed from 0 to 1, leader changed from <none> to f496bcbb6d8644ad962eceeeac3f82ca (127.0.194.193). New cstate: current_term: 1 leader_uuid: "f496bcbb6d8644ad962eceeeac3f82ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f496bcbb6d8644ad962eceeeac3f82ca" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 36187 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f20d775ac36948528aadf2fb37a3ed11" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 42655 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "737f7fce56574ef6a528cd2a48e8056d" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 39703 } health_report { overall_health: UNKNOWN } } }
I20260926 03:43:58.390969 3029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:58.384864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39786 (local address 127.0.194.193:36187)
0926 03:43:58.385068 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0926 03:43:58.385080 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 03:43:58.385247 (+ 167us) server_negotiation.cc:407] Connection header received
0926 03:43:58.385561 (+ 314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:43:58.385565 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:43:58.385650 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:43:58.385825 (+ 175us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:43:58.386197 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:58.387128 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:58.388464 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:58.388952 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:58.389107 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:43:58.389588 (+ 481us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:43:58.390176 (+ 588us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:43:58.390361 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0926 03:43:58.390441 (+ 80us) server_negotiation.cc:299] Negotiation successful
0926 03:43:58.390537 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260926 03:43:58.390969 3048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:58.384852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.193:36187 (local address 127.0.0.1:39786)
0926 03:43:58.385140 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0926 03:43:58.385159 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 03:43:58.385343 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:43:58.385854 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:43:58.385858 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:43:58.385870 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 03:43:58.386065 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:43:58.386074 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:43:58.387267 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:43:58.387274 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:43:58.388311 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:43:58.388322 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:43:58.389140 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:43:58.389146 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:43:58.389321 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:43:58.389381 (+ 60us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 03:43:58.390311 (+ 930us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 03:43:58.390335 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 03:43:58.390423 (+ 88us) client_negotiation.cc:240] Negotiation successful
0926 03:43:58.390583 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68}
I20260926 03:43:58.400756 2827 raft_consensus.cc:1261] T 3b8a073ce4d5458abdc874e9acd190b5 P f20d775ac36948528aadf2fb37a3ed11 [term 1 FOLLOWER]: Refusing update from remote peer f496bcbb6d8644ad962eceeeac3f82ca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:43:58.400856 2961 raft_consensus.cc:1261] T 3b8a073ce4d5458abdc874e9acd190b5 P 737f7fce56574ef6a528cd2a48e8056d [term 1 FOLLOWER]: Refusing update from remote peer f496bcbb6d8644ad962eceeeac3f82ca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:43:58.401852 3054 consensus_queue.cc:1035] T 3b8a073ce4d5458abdc874e9acd190b5 P f496bcbb6d8644ad962eceeeac3f82ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "f20d775ac36948528aadf2fb37a3ed11" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 42655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:43:58.402298 3038 consensus_queue.cc:1035] T 3b8a073ce4d5458abdc874e9acd190b5 P f496bcbb6d8644ad962eceeeac3f82ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "737f7fce56574ef6a528cd2a48e8056d" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 39703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:43:58.409478 3041 consensus_queue.cc:1035] T 12e1238edcbe4da891338ccef85ec9a5 P f20d775ac36948528aadf2fb37a3ed11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f496bcbb6d8644ad962eceeeac3f82ca" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 36187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:43:58.419790 3037 tablet_replica.cc:439] T 3b8a073ce4d5458abdc874e9acd190b5 P 737f7fce56574ef6a528cd2a48e8056d: TxnStatusTablet state changed. Reason: New leader f496bcbb6d8644ad962eceeeac3f82ca. Latest consensus state: current_term: 1 leader_uuid: "f496bcbb6d8644ad962eceeeac3f82ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f496bcbb6d8644ad962eceeeac3f82ca" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 36187 } } peers { permanent_uuid: "f20d775ac36948528aadf2fb37a3ed11" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 42655 } } peers { permanent_uuid: "737f7fce56574ef6a528cd2a48e8056d" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 39703 } } }
I20260926 03:43:58.420204 3037 tablet_replica.cc:442] T 3b8a073ce4d5458abdc874e9acd190b5 P 737f7fce56574ef6a528cd2a48e8056d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 03:43:58.425987 3036 tablet_replica.cc:439] T 3b8a073ce4d5458abdc874e9acd190b5 P f20d775ac36948528aadf2fb37a3ed11: TxnStatusTablet state changed. Reason: New leader f496bcbb6d8644ad962eceeeac3f82ca. Latest consensus state: current_term: 1 leader_uuid: "f496bcbb6d8644ad962eceeeac3f82ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f496bcbb6d8644ad962eceeeac3f82ca" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 36187 } } peers { permanent_uuid: "f20d775ac36948528aadf2fb37a3ed11" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 42655 } } peers { permanent_uuid: "737f7fce56574ef6a528cd2a48e8056d" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 39703 } } }
I20260926 03:43:58.426335 3036 tablet_replica.cc:442] T 3b8a073ce4d5458abdc874e9acd190b5 P f20d775ac36948528aadf2fb37a3ed11: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 03:43:58.426998 3054 tablet_replica.cc:439] T 3b8a073ce4d5458abdc874e9acd190b5 P f496bcbb6d8644ad962eceeeac3f82ca: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f496bcbb6d8644ad962eceeeac3f82ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f496bcbb6d8644ad962eceeeac3f82ca" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 36187 } } peers { permanent_uuid: "f20d775ac36948528aadf2fb37a3ed11" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 42655 } } peers { permanent_uuid: "737f7fce56574ef6a528cd2a48e8056d" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 39703 } } }
I20260926 03:43:58.427251 3054 tablet_replica.cc:442] T 3b8a073ce4d5458abdc874e9acd190b5 P f496bcbb6d8644ad962eceeeac3f82ca: This TxnStatusTablet replica's current role is: LEADER
I20260926 03:43:58.427892 3060 mvcc.cc:205] Tried to move back new op lower bound from 7333454800478539776 to 7333454800269905920. Current Snapshot: MvccSnapshot[applied={T|T < 7333454800478539776}]
I20260926 03:43:58.431891 3037 tablet_replica.cc:439] T 3b8a073ce4d5458abdc874e9acd190b5 P 737f7fce56574ef6a528cd2a48e8056d: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f496bcbb6d8644ad962eceeeac3f82ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f496bcbb6d8644ad962eceeeac3f82ca" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 36187 } } peers { permanent_uuid: "f20d775ac36948528aadf2fb37a3ed11" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 42655 } } peers { permanent_uuid: "737f7fce56574ef6a528cd2a48e8056d" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 39703 } } }
I20260926 03:43:58.432165 3037 tablet_replica.cc:442] T 3b8a073ce4d5458abdc874e9acd190b5 P 737f7fce56574ef6a528cd2a48e8056d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 03:43:58.433621 3054 tablet_replica.cc:439] T 3b8a073ce4d5458abdc874e9acd190b5 P f496bcbb6d8644ad962eceeeac3f82ca: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f496bcbb6d8644ad962eceeeac3f82ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f496bcbb6d8644ad962eceeeac3f82ca" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 36187 } } peers { permanent_uuid: "f20d775ac36948528aadf2fb37a3ed11" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 42655 } } peers { permanent_uuid: "737f7fce56574ef6a528cd2a48e8056d" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 39703 } } }
I20260926 03:43:58.433903 3054 tablet_replica.cc:442] T 3b8a073ce4d5458abdc874e9acd190b5 P f496bcbb6d8644ad962eceeeac3f82ca: This TxnStatusTablet replica's current role is: LEADER
I20260926 03:43:58.435169 3064 consensus_queue.cc:1035] T 12e1238edcbe4da891338ccef85ec9a5 P f20d775ac36948528aadf2fb37a3ed11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "737f7fce56574ef6a528cd2a48e8056d" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 39703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:43:58.452119 3064 tablet_replica.cc:439] T 3b8a073ce4d5458abdc874e9acd190b5 P f20d775ac36948528aadf2fb37a3ed11: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f496bcbb6d8644ad962eceeeac3f82ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f496bcbb6d8644ad962eceeeac3f82ca" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 36187 } } peers { permanent_uuid: "f20d775ac36948528aadf2fb37a3ed11" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 42655 } } peers { permanent_uuid: "737f7fce56574ef6a528cd2a48e8056d" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 39703 } } }
I20260926 03:43:58.452468 3064 tablet_replica.cc:442] T 3b8a073ce4d5458abdc874e9acd190b5 P f20d775ac36948528aadf2fb37a3ed11: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 03:43:58.462103 3025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:58.448757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44510 (local address 127.0.194.194:42655)
0926 03:43:58.448944 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0926 03:43:58.448950 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:43:58.455362 (+ 6412us) server_negotiation.cc:407] Connection header received
0926 03:43:58.455539 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:43:58.455545 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:43:58.455664 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:43:58.455835 (+ 171us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:43:58.456323 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:58.457534 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:58.459057 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:58.459669 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:58.459867 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:43:58.460235 (+ 368us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:43:58.461048 (+ 813us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:43:58.461213 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0926 03:43:58.461667 (+ 454us) server_negotiation.cc:299] Negotiation successful
0926 03:43:58.461761 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 03:43:58.481907 3029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:58.475067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.194:37829 (local address 127.0.194.193:36187)
0926 03:43:58.475310 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0926 03:43:58.475318 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:43:58.476056 (+ 738us) server_negotiation.cc:407] Connection header received
0926 03:43:58.476206 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:43:58.476210 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:43:58.476328 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:43:58.476426 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:43:58.477018 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:58.478076 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:58.480104 (+ 2028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:58.480509 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:58.480604 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:43:58.481162 (+ 558us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:43:58.481260 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:43:58.481324 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 03:43:58.481529 (+ 205us) server_negotiation.cc:299] Negotiation successful
0926 03:43:58.481596 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 03:43:58.482012 3074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:58.475142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.193:36187 (local address 127.0.194.194:37829)
0926 03:43:58.475872 (+ 730us) negotiation.cc:107] Waiting for socket to connect
0926 03:43:58.475897 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0926 03:43:58.476081 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:43:58.476466 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:43:58.476471 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:43:58.476483 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 03:43:58.476862 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:43:58.476874 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:43:58.478213 (+ 1339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:43:58.478219 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:43:58.479963 (+ 1744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:43:58.479978 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:43:58.480688 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:43:58.480697 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:43:58.480886 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:43:58.480965 (+ 79us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 03:43:58.481355 (+ 390us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 03:43:58.481367 (+ 12us) client_negotiation.cc:769] Sending connection context
0926 03:43:58.481452 (+ 85us) client_negotiation.cc:240] Negotiation successful
0926 03:43:58.481597 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":335,"threads_started":1}
I20260926 03:43:58.521292 3025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:58.514974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.193:55233 (local address 127.0.194.194:42655)
0926 03:43:58.515384 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0926 03:43:58.515390 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:43:58.515729 (+ 339us) server_negotiation.cc:407] Connection header received
0926 03:43:58.515940 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:43:58.515946 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:43:58.516022 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:43:58.516123 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:43:58.516667 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:58.517363 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:58.519471 (+ 2108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:58.519981 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:58.520129 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:43:58.520570 (+ 441us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:43:58.520674 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:43:58.520774 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0926 03:43:58.520931 (+ 157us) server_negotiation.cc:299] Negotiation successful
0926 03:43:58.521010 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283}
I20260926 03:43:58.521765 3078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:58.515051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.194:42655 (local address 127.0.194.193:55233)
0926 03:43:58.515623 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0926 03:43:58.515640 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 03:43:58.515809 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:43:58.516147 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:43:58.516151 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:43:58.516163 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 03:43:58.516519 (+ 356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:43:58.516532 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:43:58.517500 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:43:58.517506 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:43:58.519179 (+ 1673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:43:58.519197 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:43:58.520173 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:43:58.520183 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:43:58.520320 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:43:58.520375 (+ 55us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 03:43:58.520779 (+ 404us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 03:43:58.520793 (+ 14us) client_negotiation.cc:769] Sending connection context
0926 03:43:58.520909 (+ 116us) client_negotiation.cc:240] Negotiation successful
0926 03:43:58.521056 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":975,"threads_started":1}
I20260926 03:43:58.560056 2526 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56044:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:43:58.560397 2526 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56044:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:43:58.564132 2526 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0ccb5168dad34eb2a79efa733da1e3f6: Sending DeleteTablet for 3 replicas of tablet 12e1238edcbe4da891338ccef85ec9a5
I20260926 03:43:58.565374 2802 tablet_service.cc:1514] Processing DeleteTablet for tablet 12e1238edcbe4da891338ccef85ec9a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:43:58 UTC) from {username='kudu', principal='kudu/127.0.194.254@KRBTEST.COM'} at 127.0.0.1:44504
I20260926 03:43:58.565693 2663 tablet_service.cc:1514] Processing DeleteTablet for tablet 12e1238edcbe4da891338ccef85ec9a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:43:58 UTC) from {username='kudu', principal='kudu/127.0.194.254@KRBTEST.COM'} at 127.0.0.1:39770
I20260926 03:43:58.566496 2941 tablet_service.cc:1514] Processing DeleteTablet for tablet 12e1238edcbe4da891338ccef85ec9a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:43:58 UTC) from {username='kudu', principal='kudu/127.0.194.254@KRBTEST.COM'} at 127.0.0.1:60094
I20260926 03:43:58.575090 2526 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56044:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 03:43:58.576061 2526 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 03:43:58.578394 3081 tablet_replica.cc:330] T 12e1238edcbe4da891338ccef85ec9a5 P f20d775ac36948528aadf2fb37a3ed11: stopping tablet replica
I20260926 03:43:58.579162 3081 raft_consensus.cc:2229] T 12e1238edcbe4da891338ccef85ec9a5 P f20d775ac36948528aadf2fb37a3ed11 [term 1 LEADER]: Raft consensus shutting down.
I20260926 03:43:58.580122 3081 raft_consensus.cc:2258] T 12e1238edcbe4da891338ccef85ec9a5 P f20d775ac36948528aadf2fb37a3ed11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:43:58.582803 3081 ts_tablet_manager.cc:1905] T 12e1238edcbe4da891338ccef85ec9a5 P f20d775ac36948528aadf2fb37a3ed11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:43:58.591687 3081 ts_tablet_manager.cc:1918] T 12e1238edcbe4da891338ccef85ec9a5 P f20d775ac36948528aadf2fb37a3ed11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 03:43:58.592010 3081 log.cc:1196] T 12e1238edcbe4da891338ccef85ec9a5 P f20d775ac36948528aadf2fb37a3ed11: Deleting WAL directory at /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790394221564982-779-0/minicluster-data/ts-1/wal/wals/12e1238edcbe4da891338ccef85ec9a5
I20260926 03:43:58.592762 3081 ts_tablet_manager.cc:1939] T 12e1238edcbe4da891338ccef85ec9a5 P f20d775ac36948528aadf2fb37a3ed11: Deleting consensus metadata
I20260926 03:43:58.594635 2802 tablet_service.cc:1467] Processing CreateTablet for tablet 65d2586e39214d408ba69ef5727b26cd (DEFAULT_TABLE table=test-table [id=59c121f2e9c74f6c925240d593d0228b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:43:58.595132 2802 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65d2586e39214d408ba69ef5727b26cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:43:58.595386 2662 tablet_service.cc:1467] Processing CreateTablet for tablet 65d2586e39214d408ba69ef5727b26cd (DEFAULT_TABLE table=test-table [id=59c121f2e9c74f6c925240d593d0228b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:43:58.595860 2662 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65d2586e39214d408ba69ef5727b26cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:43:58.599330 2940 tablet_service.cc:1467] Processing CreateTablet for tablet 65d2586e39214d408ba69ef5727b26cd (DEFAULT_TABLE table=test-table [id=59c121f2e9c74f6c925240d593d0228b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:43:58.599880 2940 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65d2586e39214d408ba69ef5727b26cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:43:58.608255 3083 tablet_replica.cc:330] T 12e1238edcbe4da891338ccef85ec9a5 P 737f7fce56574ef6a528cd2a48e8056d: stopping tablet replica
I20260926 03:43:58.608816 3083 raft_consensus.cc:2229] T 12e1238edcbe4da891338ccef85ec9a5 P 737f7fce56574ef6a528cd2a48e8056d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:43:58.609282 3083 raft_consensus.cc:2258] T 12e1238edcbe4da891338ccef85ec9a5 P 737f7fce56574ef6a528cd2a48e8056d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:43:58.610199 3082 tablet_replica.cc:330] T 12e1238edcbe4da891338ccef85ec9a5 P f496bcbb6d8644ad962eceeeac3f82ca: stopping tablet replica
I20260926 03:43:58.611577 3082 raft_consensus.cc:2229] T 12e1238edcbe4da891338ccef85ec9a5 P f496bcbb6d8644ad962eceeeac3f82ca [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:43:58.611971 3082 raft_consensus.cc:2258] T 12e1238edcbe4da891338ccef85ec9a5 P f496bcbb6d8644ad962eceeeac3f82ca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:43:58.613409 3083 ts_tablet_manager.cc:1905] T 12e1238edcbe4da891338ccef85ec9a5 P 737f7fce56574ef6a528cd2a48e8056d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:43:58.615743 3084 tablet_bootstrap.cc:492] T 65d2586e39214d408ba69ef5727b26cd P f496bcbb6d8644ad962eceeeac3f82ca: Bootstrap starting.
I20260926 03:43:58.617684 3084 tablet_bootstrap.cc:654] T 65d2586e39214d408ba69ef5727b26cd P f496bcbb6d8644ad962eceeeac3f82ca: Neither blocks nor log segments found. Creating new log.
I20260926 03:43:58.620683 3082 ts_tablet_manager.cc:1905] T 12e1238edcbe4da891338ccef85ec9a5 P f496bcbb6d8644ad962eceeeac3f82ca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:43:58.621917 2513 catalog_manager.cc:4841] TS f20d775ac36948528aadf2fb37a3ed11 (127.0.194.194:42655): tablet 12e1238edcbe4da891338ccef85ec9a5 (table test-table [id=7bfddf5e2b6644059aaf1f446942b200]) successfully deleted
I20260926 03:43:58.628190 3084 tablet_bootstrap.cc:492] T 65d2586e39214d408ba69ef5727b26cd P f496bcbb6d8644ad962eceeeac3f82ca: No bootstrap required, opened a new log
I20260926 03:43:58.628376 3084 ts_tablet_manager.cc:1397] T 65d2586e39214d408ba69ef5727b26cd P f496bcbb6d8644ad962eceeeac3f82ca: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.000s
I20260926 03:43:58.629189 3084 raft_consensus.cc:348] T 65d2586e39214d408ba69ef5727b26cd P f496bcbb6d8644ad962eceeeac3f82ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f496bcbb6d8644ad962eceeeac3f82ca" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 36187 } } peers { permanent_uuid: "f20d775ac36948528aadf2fb37a3ed11" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 42655 } } peers { permanent_uuid: "737f7fce56574ef6a528cd2a48e8056d" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 39703 } }
I20260926 03:43:58.629494 3084 raft_consensus.cc:374] T 65d2586e39214d408ba69ef5727b26cd P f496bcbb6d8644ad962eceeeac3f82ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:43:58.629593 3084 raft_consensus.cc:729] T 65d2586e39214d408ba69ef5727b26cd P f496bcbb6d8644ad962eceeeac3f82ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f496bcbb6d8644ad962eceeeac3f82ca, State: Initialized, Role: FOLLOWER
I20260926 03:43:58.629876 3084 consensus_queue.cc:260] T 65d2586e39214d408ba69ef5727b26cd P f496bcbb6d8644ad962eceeeac3f82ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f496bcbb6d8644ad962eceeeac3f82ca" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 36187 } } peers { permanent_uuid: "f20d775ac36948528aadf2fb37a3ed11" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 42655 } } peers { permanent_uuid: "737f7fce56574ef6a528cd2a48e8056d" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 39703 } }
I20260926 03:43:58.630944 3083 ts_tablet_manager.cc:1918] T 12e1238edcbe4da891338ccef85ec9a5 P 737f7fce56574ef6a528cd2a48e8056d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 03:43:58.631156 3083 log.cc:1196] T 12e1238edcbe4da891338ccef85ec9a5 P 737f7fce56574ef6a528cd2a48e8056d: Deleting WAL directory at /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790394221564982-779-0/minicluster-data/ts-2/wal/wals/12e1238edcbe4da891338ccef85ec9a5
I20260926 03:43:58.633124 3087 tablet_bootstrap.cc:492] T 65d2586e39214d408ba69ef5727b26cd P 737f7fce56574ef6a528cd2a48e8056d: Bootstrap starting.
I20260926 03:43:58.633214 3084 ts_tablet_manager.cc:1428] T 65d2586e39214d408ba69ef5727b26cd P f496bcbb6d8644ad962eceeeac3f82ca: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20260926 03:43:58.634066 3082 ts_tablet_manager.cc:1918] T 12e1238edcbe4da891338ccef85ec9a5 P f496bcbb6d8644ad962eceeeac3f82ca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 03:43:58.634243 3082 log.cc:1196] T 12e1238edcbe4da891338ccef85ec9a5 P f496bcbb6d8644ad962eceeeac3f82ca: Deleting WAL directory at /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790394221564982-779-0/minicluster-data/ts-0/wal/wals/12e1238edcbe4da891338ccef85ec9a5
I20260926 03:43:58.635325 3083 ts_tablet_manager.cc:1939] T 12e1238edcbe4da891338ccef85ec9a5 P 737f7fce56574ef6a528cd2a48e8056d: Deleting consensus metadata
I20260926 03:43:58.635931 3086 tablet_bootstrap.cc:492] T 65d2586e39214d408ba69ef5727b26cd P f20d775ac36948528aadf2fb37a3ed11: Bootstrap starting.
I20260926 03:43:58.636152 3082 ts_tablet_manager.cc:1939] T 12e1238edcbe4da891338ccef85ec9a5 P f496bcbb6d8644ad962eceeeac3f82ca: Deleting consensus metadata
I20260926 03:43:58.637398 3087 tablet_bootstrap.cc:654] T 65d2586e39214d408ba69ef5727b26cd P 737f7fce56574ef6a528cd2a48e8056d: Neither blocks nor log segments found. Creating new log.
I20260926 03:43:58.637354 2510 catalog_manager.cc:4841] TS 737f7fce56574ef6a528cd2a48e8056d (127.0.194.195:39703): tablet 12e1238edcbe4da891338ccef85ec9a5 (table test-table [id=7bfddf5e2b6644059aaf1f446942b200]) successfully deleted
I20260926 03:43:58.637745 2513 catalog_manager.cc:4841] TS f496bcbb6d8644ad962eceeeac3f82ca (127.0.194.193:36187): tablet 12e1238edcbe4da891338ccef85ec9a5 (table test-table [id=7bfddf5e2b6644059aaf1f446942b200]) successfully deleted
I20260926 03:43:58.639957 3086 tablet_bootstrap.cc:654] T 65d2586e39214d408ba69ef5727b26cd P f20d775ac36948528aadf2fb37a3ed11: Neither blocks nor log segments found. Creating new log.
I20260926 03:43:58.647341 3087 tablet_bootstrap.cc:492] T 65d2586e39214d408ba69ef5727b26cd P 737f7fce56574ef6a528cd2a48e8056d: No bootstrap required, opened a new log
I20260926 03:43:58.647658 3087 ts_tablet_manager.cc:1397] T 65d2586e39214d408ba69ef5727b26cd P 737f7fce56574ef6a528cd2a48e8056d: Time spent bootstrapping tablet: real 0.015s user 0.007s sys 0.003s
I20260926 03:43:58.648835 3087 raft_consensus.cc:348] T 65d2586e39214d408ba69ef5727b26cd P 737f7fce56574ef6a528cd2a48e8056d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f496bcbb6d8644ad962eceeeac3f82ca" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 36187 } } peers { permanent_uuid: "f20d775ac36948528aadf2fb37a3ed11" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 42655 } } peers { permanent_uuid: "737f7fce56574ef6a528cd2a48e8056d" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 39703 } }
I20260926 03:43:58.649282 3087 raft_consensus.cc:374] T 65d2586e39214d408ba69ef5727b26cd P 737f7fce56574ef6a528cd2a48e8056d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:43:58.649446 3087 raft_consensus.cc:729] T 65d2586e39214d408ba69ef5727b26cd P 737f7fce56574ef6a528cd2a48e8056d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 737f7fce56574ef6a528cd2a48e8056d, State: Initialized, Role: FOLLOWER
I20260926 03:43:58.649910 3087 consensus_queue.cc:260] T 65d2586e39214d408ba69ef5727b26cd P 737f7fce56574ef6a528cd2a48e8056d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f496bcbb6d8644ad962eceeeac3f82ca" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 36187 } } peers { permanent_uuid: "f20d775ac36948528aadf2fb37a3ed11" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 42655 } } peers { permanent_uuid: "737f7fce56574ef6a528cd2a48e8056d" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 39703 } }
I20260926 03:43:58.653584 3087 ts_tablet_manager.cc:1428] T 65d2586e39214d408ba69ef5727b26cd P 737f7fce56574ef6a528cd2a48e8056d: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20260926 03:43:58.658185 3086 tablet_bootstrap.cc:492] T 65d2586e39214d408ba69ef5727b26cd P f20d775ac36948528aadf2fb37a3ed11: No bootstrap required, opened a new log
I20260926 03:43:58.658389 3086 ts_tablet_manager.cc:1397] T 65d2586e39214d408ba69ef5727b26cd P f20d775ac36948528aadf2fb37a3ed11: Time spent bootstrapping tablet: real 0.023s user 0.011s sys 0.005s
I20260926 03:43:58.659191 3086 raft_consensus.cc:348] T 65d2586e39214d408ba69ef5727b26cd P f20d775ac36948528aadf2fb37a3ed11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f496bcbb6d8644ad962eceeeac3f82ca" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 36187 } } peers { permanent_uuid: "f20d775ac36948528aadf2fb37a3ed11" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 42655 } } peers { permanent_uuid: "737f7fce56574ef6a528cd2a48e8056d" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 39703 } }
I20260926 03:43:58.659453 3086 raft_consensus.cc:374] T 65d2586e39214d408ba69ef5727b26cd P f20d775ac36948528aadf2fb37a3ed11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:43:58.659580 3086 raft_consensus.cc:729] T 65d2586e39214d408ba69ef5727b26cd P f20d775ac36948528aadf2fb37a3ed11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f20d775ac36948528aadf2fb37a3ed11, State: Initialized, Role: FOLLOWER
I20260926 03:43:58.659879 3086 consensus_queue.cc:260] T 65d2586e39214d408ba69ef5727b26cd P f20d775ac36948528aadf2fb37a3ed11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f496bcbb6d8644ad962eceeeac3f82ca" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 36187 } } peers { permanent_uuid: "f20d775ac36948528aadf2fb37a3ed11" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 42655 } } peers { permanent_uuid: "737f7fce56574ef6a528cd2a48e8056d" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 39703 } }
I20260926 03:43:58.660533 3086 ts_tablet_manager.cc:1428] T 65d2586e39214d408ba69ef5727b26cd P f20d775ac36948528aadf2fb37a3ed11: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260926 03:43:58.664180 3037 raft_consensus.cc:482] T 65d2586e39214d408ba69ef5727b26cd P 737f7fce56574ef6a528cd2a48e8056d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 03:43:58.664383 3037 raft_consensus.cc:504] T 65d2586e39214d408ba69ef5727b26cd P 737f7fce56574ef6a528cd2a48e8056d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f496bcbb6d8644ad962eceeeac3f82ca" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 36187 } } peers { permanent_uuid: "f20d775ac36948528aadf2fb37a3ed11" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 42655 } } peers { permanent_uuid: "737f7fce56574ef6a528cd2a48e8056d" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 39703 } }
I20260926 03:43:58.666136 3037 leader_election.cc:288] T 65d2586e39214d408ba69ef5727b26cd P 737f7fce56574ef6a528cd2a48e8056d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f496bcbb6d8644ad962eceeeac3f82ca (127.0.194.193:36187), f20d775ac36948528aadf2fb37a3ed11 (127.0.194.194:42655)
I20260926 03:43:58.680478 3029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:58.666794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.195:54689 (local address 127.0.194.193:36187)
0926 03:43:58.667042 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0926 03:43:58.667047 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:43:58.667068 (+ 21us) server_negotiation.cc:407] Connection header received
0926 03:43:58.667281 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:43:58.667286 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:43:58.667397 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:43:58.667468 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:43:58.669006 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:58.670575 (+ 1569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:58.674086 (+ 3511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:58.676057 (+ 1971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:58.676273 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:43:58.676391 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0926 03:43:58.679792 (+ 3401us) server_negotiation.cc:299] Negotiation successful
0926 03:43:58.680023 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 03:43:58.681444 3090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:58.666331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.193:36187 (local address 127.0.194.195:54689)
0926 03:43:58.666891 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0926 03:43:58.666914 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 03:43:58.667051 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:43:58.667719 (+ 668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:43:58.667723 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:43:58.667750 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:43:58.668769 (+ 1019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:43:58.668788 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:43:58.670841 (+ 2053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:43:58.670849 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:43:58.673829 (+ 2980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:43:58.673851 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:43:58.678662 (+ 4811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:43:58.678672 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:43:58.678993 (+ 321us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:43:58.679046 (+ 53us) client_negotiation.cc:769] Sending connection context
0926 03:43:58.680704 (+ 1658us) client_negotiation.cc:240] Negotiation successful
0926 03:43:58.680870 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":811,"threads_started":1}
I20260926 03:43:58.683104 2687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d2586e39214d408ba69ef5727b26cd" candidate_uuid: "737f7fce56574ef6a528cd2a48e8056d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f496bcbb6d8644ad962eceeeac3f82ca" is_pre_election: true
I20260926 03:43:58.683430 2687 raft_consensus.cc:2454] T 65d2586e39214d408ba69ef5727b26cd P f496bcbb6d8644ad962eceeeac3f82ca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 737f7fce56574ef6a528cd2a48e8056d in term 0.
I20260926 03:43:58.684173 2895 leader_election.cc:302] T 65d2586e39214d408ba69ef5727b26cd P 737f7fce56574ef6a528cd2a48e8056d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 737f7fce56574ef6a528cd2a48e8056d, f496bcbb6d8644ad962eceeeac3f82ca; no voters:
I20260926 03:43:58.684643 3037 raft_consensus.cc:2781] T 65d2586e39214d408ba69ef5727b26cd P 737f7fce56574ef6a528cd2a48e8056d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 03:43:58.684769 3037 raft_consensus.cc:482] T 65d2586e39214d408ba69ef5727b26cd P 737f7fce56574ef6a528cd2a48e8056d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 03:43:58.684872 3037 raft_consensus.cc:3037] T 65d2586e39214d408ba69ef5727b26cd P 737f7fce56574ef6a528cd2a48e8056d [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:43:58.688329 3037 raft_consensus.cc:504] T 65d2586e39214d408ba69ef5727b26cd P 737f7fce56574ef6a528cd2a48e8056d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f496bcbb6d8644ad962eceeeac3f82ca" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 36187 } } peers { permanent_uuid: "f20d775ac36948528aadf2fb37a3ed11" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 42655 } } peers { permanent_uuid: "737f7fce56574ef6a528cd2a48e8056d" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 39703 } }
I20260926 03:43:58.689139 3037 leader_election.cc:288] T 65d2586e39214d408ba69ef5727b26cd P 737f7fce56574ef6a528cd2a48e8056d [CANDIDATE]: Term 1 election: Requested vote from peers f496bcbb6d8644ad962eceeeac3f82ca (127.0.194.193:36187), f20d775ac36948528aadf2fb37a3ed11 (127.0.194.194:42655)
I20260926 03:43:58.689734 3025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:58.667905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.195:44589 (local address 127.0.194.194:42655)
0926 03:43:58.668127 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0926 03:43:58.668135 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:43:58.681837 (+ 13702us) server_negotiation.cc:407] Connection header received
0926 03:43:58.682562 (+ 725us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:43:58.682570 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:43:58.682658 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:43:58.682724 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:43:58.683452 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:58.684593 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:58.687069 (+ 2476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:58.688529 (+ 1460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:58.688677 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:43:58.688747 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 03:43:58.689257 (+ 510us) server_negotiation.cc:299] Negotiation successful
0926 03:43:58.689403 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260926 03:43:58.689937 2687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d2586e39214d408ba69ef5727b26cd" candidate_uuid: "737f7fce56574ef6a528cd2a48e8056d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f496bcbb6d8644ad962eceeeac3f82ca"
I20260926 03:43:58.690186 2687 raft_consensus.cc:3037] T 65d2586e39214d408ba69ef5727b26cd P f496bcbb6d8644ad962eceeeac3f82ca [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:43:58.690435 3090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:58.667516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.194:42655 (local address 127.0.194.195:44589)
0926 03:43:58.681718 (+ 14202us) negotiation.cc:107] Waiting for socket to connect
0926 03:43:58.681742 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 03:43:58.681979 (+ 237us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:43:58.682943 (+ 964us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:43:58.682947 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:43:58.682975 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:43:58.683146 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:43:58.683158 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:43:58.684880 (+ 1722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:43:58.684889 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:43:58.686878 (+ 1989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:43:58.686907 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:43:58.688884 (+ 1977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:43:58.688889 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:43:58.689055 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:43:58.689090 (+ 35us) client_negotiation.cc:769] Sending connection context
0926 03:43:58.689979 (+ 889us) client_negotiation.cc:240] Negotiation successful
0926 03:43:58.690129 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14099,"thread_start_us":14733,"threads_started":1}
I20260926 03:43:58.691509 2826 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d2586e39214d408ba69ef5727b26cd" candidate_uuid: "737f7fce56574ef6a528cd2a48e8056d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f20d775ac36948528aadf2fb37a3ed11" is_pre_election: true
I20260926 03:43:58.691808 2826 raft_consensus.cc:2454] T 65d2586e39214d408ba69ef5727b26cd P f20d775ac36948528aadf2fb37a3ed11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 737f7fce56574ef6a528cd2a48e8056d in term 0.
I20260926 03:43:58.692518 2826 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d2586e39214d408ba69ef5727b26cd" candidate_uuid: "737f7fce56574ef6a528cd2a48e8056d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f20d775ac36948528aadf2fb37a3ed11"
I20260926 03:43:58.692718 2826 raft_consensus.cc:3037] T 65d2586e39214d408ba69ef5727b26cd P f20d775ac36948528aadf2fb37a3ed11 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:43:58.693562 2687 raft_consensus.cc:2454] T 65d2586e39214d408ba69ef5727b26cd P f496bcbb6d8644ad962eceeeac3f82ca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 737f7fce56574ef6a528cd2a48e8056d in term 1.
I20260926 03:43:58.695731 2826 raft_consensus.cc:2454] T 65d2586e39214d408ba69ef5727b26cd P f20d775ac36948528aadf2fb37a3ed11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 737f7fce56574ef6a528cd2a48e8056d in term 1.
I20260926 03:43:58.696533 2895 leader_election.cc:302] T 65d2586e39214d408ba69ef5727b26cd P 737f7fce56574ef6a528cd2a48e8056d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 737f7fce56574ef6a528cd2a48e8056d, f496bcbb6d8644ad962eceeeac3f82ca; no voters:
I20260926 03:43:58.696861 3037 raft_consensus.cc:2781] T 65d2586e39214d408ba69ef5727b26cd P 737f7fce56574ef6a528cd2a48e8056d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:43:58.699081 3037 raft_consensus.cc:686] T 65d2586e39214d408ba69ef5727b26cd P 737f7fce56574ef6a528cd2a48e8056d [term 1 LEADER]: Becoming Leader. State: Replica: 737f7fce56574ef6a528cd2a48e8056d, State: Running, Role: LEADER
I20260926 03:43:58.699599 3037 consensus_queue.cc:237] T 65d2586e39214d408ba69ef5727b26cd P 737f7fce56574ef6a528cd2a48e8056d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f496bcbb6d8644ad962eceeeac3f82ca" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 36187 } } peers { permanent_uuid: "f20d775ac36948528aadf2fb37a3ed11" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 42655 } } peers { permanent_uuid: "737f7fce56574ef6a528cd2a48e8056d" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 39703 } }
I20260926 03:43:58.703320 2524 catalog_manager.cc:5495] T 65d2586e39214d408ba69ef5727b26cd P 737f7fce56574ef6a528cd2a48e8056d reported cstate change: term changed from 0 to 1, leader changed from <none> to 737f7fce56574ef6a528cd2a48e8056d (127.0.194.195). New cstate: current_term: 1 leader_uuid: "737f7fce56574ef6a528cd2a48e8056d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f496bcbb6d8644ad962eceeeac3f82ca" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 36187 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f20d775ac36948528aadf2fb37a3ed11" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 42655 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "737f7fce56574ef6a528cd2a48e8056d" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 39703 } health_report { overall_health: HEALTHY } } }
I20260926 03:43:58.737749 3027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:58.732462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60100 (local address 127.0.194.195:39703)
0926 03:43:58.732615 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0926 03:43:58.732623 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:43:58.732836 (+ 213us) server_negotiation.cc:407] Connection header received
0926 03:43:58.732920 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:43:58.732924 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:43:58.733004 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:43:58.733161 (+ 157us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:43:58.733554 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:58.734320 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:58.735672 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:58.736104 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:58.736235 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:43:58.736620 (+ 385us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:43:58.737152 (+ 532us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:43:58.737309 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0926 03:43:58.737377 (+ 68us) server_negotiation.cc:299] Negotiation successful
0926 03:43:58.737465 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 03:43:58.753664 3029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:58.748086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.195:47745 (local address 127.0.194.193:36187)
0926 03:43:58.748494 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0926 03:43:58.748506 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 03:43:58.748843 (+ 337us) server_negotiation.cc:407] Connection header received
0926 03:43:58.749072 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:43:58.749081 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:43:58.749151 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:43:58.749246 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:43:58.749724 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:58.750482 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:58.751863 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:58.752320 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:58.752472 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:43:58.752896 (+ 424us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:43:58.753017 (+ 121us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:43:58.753133 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0926 03:43:58.753276 (+ 143us) server_negotiation.cc:299] Negotiation successful
0926 03:43:58.753364 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":241}
I20260926 03:43:58.754006 3096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:58.748099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.193:36187 (local address 127.0.194.195:47745)
0926 03:43:58.748731 (+ 632us) negotiation.cc:107] Waiting for socket to connect
0926 03:43:58.748761 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0926 03:43:58.748930 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:43:58.749260 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:43:58.749264 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:43:58.749276 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 03:43:58.749564 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:43:58.749573 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:43:58.750611 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:43:58.750620 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:43:58.751721 (+ 1101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:43:58.751733 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:43:58.752505 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:43:58.752511 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:43:58.752672 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:43:58.752723 (+ 51us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 03:43:58.753152 (+ 429us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 03:43:58.753162 (+ 10us) client_negotiation.cc:769] Sending connection context
0926 03:43:58.753257 (+ 95us) client_negotiation.cc:240] Negotiation successful
0926 03:43:58.753402 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":422,"threads_started":1}
I20260926 03:43:58.763972 2687 raft_consensus.cc:1261] T 65d2586e39214d408ba69ef5727b26cd P f496bcbb6d8644ad962eceeeac3f82ca [term 1 FOLLOWER]: Refusing update from remote peer 737f7fce56574ef6a528cd2a48e8056d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:43:58.764724 3092 consensus_queue.cc:1035] T 65d2586e39214d408ba69ef5727b26cd P 737f7fce56574ef6a528cd2a48e8056d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f496bcbb6d8644ad962eceeeac3f82ca" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 36187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:43:58.765662 2826 raft_consensus.cc:1261] T 65d2586e39214d408ba69ef5727b26cd P f20d775ac36948528aadf2fb37a3ed11 [term 1 FOLLOWER]: Refusing update from remote peer 737f7fce56574ef6a528cd2a48e8056d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:43:58.766947 3092 consensus_queue.cc:1035] T 65d2586e39214d408ba69ef5727b26cd P 737f7fce56574ef6a528cd2a48e8056d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f20d775ac36948528aadf2fb37a3ed11" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 42655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:43:58.773480 3059 mvcc.cc:205] Tried to move back new op lower bound from 7333454801970434048 to 7333454801717886976. Current Snapshot: MvccSnapshot[applied={T|T < 7333454801970434048}]
I20260926 03:43:58.783739 3061 mvcc.cc:205] Tried to move back new op lower bound from 7333454801970434048 to 7333454801717886976. Current Snapshot: MvccSnapshot[applied={T|T < 7333454801970434048}]
I20260926 03:43:58.817299 3078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:58.809166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.195:39703 (local address 127.0.194.193:35155)
0926 03:43:58.809523 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0926 03:43:58.809549 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0926 03:43:58.809638 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:43:58.810041 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:43:58.810045 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:43:58.810057 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 03:43:58.810289 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:43:58.810305 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:43:58.811674 (+ 1369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:43:58.811681 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:43:58.813270 (+ 1589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:43:58.813284 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:43:58.816038 (+ 2754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:43:58.816044 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:43:58.816220 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:43:58.816265 (+ 45us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 03:43:58.816715 (+ 450us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 03:43:58.816726 (+ 11us) client_negotiation.cc:769] Sending connection context
0926 03:43:58.816798 (+ 72us) client_negotiation.cc:240] Negotiation successful
0926 03:43:58.816969 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":215}
I20260926 03:43:58.817879 3027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:58.809448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.193:35155 (local address 127.0.194.195:39703)
0926 03:43:58.809805 (+ 357us) server_negotiation.cc:206] Beginning negotiation
0926 03:43:58.809809 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:43:58.809821 (+ 12us) server_negotiation.cc:407] Connection header received
0926 03:43:58.809870 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:43:58.809873 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:43:58.809929 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:43:58.810026 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:43:58.810642 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:58.811526 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:58.815329 (+ 3803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:58.815887 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:58.816027 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:43:58.816457 (+ 430us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:43:58.816588 (+ 131us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:43:58.816666 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 03:43:58.817526 (+ 860us) server_negotiation.cc:299] Negotiation successful
0926 03:43:58.817604 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261}
W20260926 03:43:58.819322 3060 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260926 03:44:00.491096 2524 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56044:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:44:00.491375 2524 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56044:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:44:00.495285 2524 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0ccb5168dad34eb2a79efa733da1e3f6: Sending DeleteTablet for 3 replicas of tablet 65d2586e39214d408ba69ef5727b26cd
I20260926 03:44:00.496097 2663 tablet_service.cc:1514] Processing DeleteTablet for tablet 65d2586e39214d408ba69ef5727b26cd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:44:00 UTC) from {username='kudu', principal='kudu/127.0.194.254@KRBTEST.COM'} at 127.0.0.1:39770
I20260926 03:44:00.496228 2802 tablet_service.cc:1514] Processing DeleteTablet for tablet 65d2586e39214d408ba69ef5727b26cd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:44:00 UTC) from {username='kudu', principal='kudu/127.0.194.254@KRBTEST.COM'} at 127.0.0.1:44504
I20260926 03:44:00.496233 2940 tablet_service.cc:1514] Processing DeleteTablet for tablet 65d2586e39214d408ba69ef5727b26cd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:44:00 UTC) from {username='kudu', principal='kudu/127.0.194.254@KRBTEST.COM'} at 127.0.0.1:60094
I20260926 03:44:00.509832 3125 tablet_replica.cc:330] T 65d2586e39214d408ba69ef5727b26cd P f496bcbb6d8644ad962eceeeac3f82ca: stopping tablet replica
I20260926 03:44:00.510067 3126 tablet_replica.cc:330] T 65d2586e39214d408ba69ef5727b26cd P f20d775ac36948528aadf2fb37a3ed11: stopping tablet replica
I20260926 03:44:00.510236 3125 raft_consensus.cc:2229] T 65d2586e39214d408ba69ef5727b26cd P f496bcbb6d8644ad962eceeeac3f82ca [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:44:00.510329 3126 raft_consensus.cc:2229] T 65d2586e39214d408ba69ef5727b26cd P f20d775ac36948528aadf2fb37a3ed11 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:44:00.510578 3125 raft_consensus.cc:2258] T 65d2586e39214d408ba69ef5727b26cd P f496bcbb6d8644ad962eceeeac3f82ca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:44:00.510577 3126 raft_consensus.cc:2258] T 65d2586e39214d408ba69ef5727b26cd P f20d775ac36948528aadf2fb37a3ed11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:44:00.511718 3126 ts_tablet_manager.cc:1905] T 65d2586e39214d408ba69ef5727b26cd P f20d775ac36948528aadf2fb37a3ed11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:44:00.511718 3125 ts_tablet_manager.cc:1905] T 65d2586e39214d408ba69ef5727b26cd P f496bcbb6d8644ad962eceeeac3f82ca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:44:00.518560 3125 ts_tablet_manager.cc:1918] T 65d2586e39214d408ba69ef5727b26cd P f496bcbb6d8644ad962eceeeac3f82ca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 03:44:00.518579 3126 ts_tablet_manager.cc:1918] T 65d2586e39214d408ba69ef5727b26cd P f20d775ac36948528aadf2fb37a3ed11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 03:44:00.518760 3125 log.cc:1196] T 65d2586e39214d408ba69ef5727b26cd P f496bcbb6d8644ad962eceeeac3f82ca: Deleting WAL directory at /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790394221564982-779-0/minicluster-data/ts-0/wal/wals/65d2586e39214d408ba69ef5727b26cd
I20260926 03:44:00.519282 3125 ts_tablet_manager.cc:1939] T 65d2586e39214d408ba69ef5727b26cd P f496bcbb6d8644ad962eceeeac3f82ca: Deleting consensus metadata
I20260926 03:44:00.519320 3127 tablet_replica.cc:330] T 65d2586e39214d408ba69ef5727b26cd P 737f7fce56574ef6a528cd2a48e8056d: stopping tablet replica
I20260926 03:44:00.519533 3127 raft_consensus.cc:2229] T 65d2586e39214d408ba69ef5727b26cd P 737f7fce56574ef6a528cd2a48e8056d [term 1 LEADER]: Raft consensus shutting down.
I20260926 03:44:00.519603 3126 log.cc:1196] T 65d2586e39214d408ba69ef5727b26cd P f20d775ac36948528aadf2fb37a3ed11: Deleting WAL directory at /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790394221564982-779-0/minicluster-data/ts-1/wal/wals/65d2586e39214d408ba69ef5727b26cd
I20260926 03:44:00.520115 3127 raft_consensus.cc:2258] T 65d2586e39214d408ba69ef5727b26cd P 737f7fce56574ef6a528cd2a48e8056d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:44:00.520192 3126 ts_tablet_manager.cc:1939] T 65d2586e39214d408ba69ef5727b26cd P f20d775ac36948528aadf2fb37a3ed11: Deleting consensus metadata
I20260926 03:44:00.520500 2513 catalog_manager.cc:4841] TS f496bcbb6d8644ad962eceeeac3f82ca (127.0.194.193:36187): tablet 65d2586e39214d408ba69ef5727b26cd (table test-table [id=59c121f2e9c74f6c925240d593d0228b]) successfully deleted
I20260926 03:44:00.521368 3127 ts_tablet_manager.cc:1905] T 65d2586e39214d408ba69ef5727b26cd P 737f7fce56574ef6a528cd2a48e8056d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:44:00.521715 2513 catalog_manager.cc:4841] TS f20d775ac36948528aadf2fb37a3ed11 (127.0.194.194:42655): tablet 65d2586e39214d408ba69ef5727b26cd (table test-table [id=59c121f2e9c74f6c925240d593d0228b]) successfully deleted
I20260926 03:44:00.528752 3127 ts_tablet_manager.cc:1918] T 65d2586e39214d408ba69ef5727b26cd P 737f7fce56574ef6a528cd2a48e8056d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 03:44:00.528909 3127 log.cc:1196] T 65d2586e39214d408ba69ef5727b26cd P 737f7fce56574ef6a528cd2a48e8056d: Deleting WAL directory at /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790394221564982-779-0/minicluster-data/ts-2/wal/wals/65d2586e39214d408ba69ef5727b26cd
I20260926 03:44:00.529317 3127 ts_tablet_manager.cc:1939] T 65d2586e39214d408ba69ef5727b26cd P 737f7fce56574ef6a528cd2a48e8056d: Deleting consensus metadata
I20260926 03:44:00.530426 2510 catalog_manager.cc:4841] TS 737f7fce56574ef6a528cd2a48e8056d (127.0.194.195:39703): tablet 65d2586e39214d408ba69ef5727b26cd (table test-table [id=59c121f2e9c74f6c925240d593d0228b]) successfully deleted
Sep 26 03:44:00 dist-test-slave-n2th krb5kdc[2481](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394240, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 03:44:00 dist-test-slave-n2th krb5kdc[2481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394240, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.194.254@KRBTEST.COM
I20260926 03:44:00.558712 3135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:00.545088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56064 (local address 127.0.194.254:43979)
0926 03:44:00.545645 (+ 557us) server_negotiation.cc:206] Beginning negotiation
0926 03:44:00.545649 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:44:00.545795 (+ 146us) server_negotiation.cc:407] Connection header received
0926 03:44:00.545945 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:44:00.545949 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:44:00.546012 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:44:00.546118 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:44:00.547490 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:00.548263 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:00.549505 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:00.549915 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:00.550019 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:00.553885 (+ 3866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:44:00.553890 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:44:00.553893 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:44:00.553926 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:44:00.556453 (+ 2527us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:00.557060 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:00.557064 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:00.557066 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:00.557185 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:00.557500 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:00.557503 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:00.557505 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:00.557773 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:44:00.557949 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0926 03:44:00.558178 (+ 229us) server_negotiation.cc:299] Negotiation successful
0926 03:44:00.558432 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":451,"thread_start_us":306,"threads_started":1}
I20260926 03:44:00.562243 2524 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56064:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 03:44:00.562911 2524 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 03:44:00.569181 2663 tablet_service.cc:1467] Processing CreateTablet for tablet e315dbe4cfeb4c23aaa910a6cbda3339 (DEFAULT_TABLE table=test-table [id=5084c26562b545ba9b07c5994becb632]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:44:00.569293 2802 tablet_service.cc:1467] Processing CreateTablet for tablet e315dbe4cfeb4c23aaa910a6cbda3339 (DEFAULT_TABLE table=test-table [id=5084c26562b545ba9b07c5994becb632]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:44:00.569445 2940 tablet_service.cc:1467] Processing CreateTablet for tablet e315dbe4cfeb4c23aaa910a6cbda3339 (DEFAULT_TABLE table=test-table [id=5084c26562b545ba9b07c5994becb632]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:44:00.569679 2663 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e315dbe4cfeb4c23aaa910a6cbda3339. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:44:00.569779 2802 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e315dbe4cfeb4c23aaa910a6cbda3339. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:44:00.569872 2940 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e315dbe4cfeb4c23aaa910a6cbda3339. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:44:00.577548 3138 tablet_bootstrap.cc:492] T e315dbe4cfeb4c23aaa910a6cbda3339 P 737f7fce56574ef6a528cd2a48e8056d: Bootstrap starting.
I20260926 03:44:00.577634 3136 tablet_bootstrap.cc:492] T e315dbe4cfeb4c23aaa910a6cbda3339 P f20d775ac36948528aadf2fb37a3ed11: Bootstrap starting.
I20260926 03:44:00.579185 3138 tablet_bootstrap.cc:654] T e315dbe4cfeb4c23aaa910a6cbda3339 P 737f7fce56574ef6a528cd2a48e8056d: Neither blocks nor log segments found. Creating new log.
I20260926 03:44:00.580008 3136 tablet_bootstrap.cc:654] T e315dbe4cfeb4c23aaa910a6cbda3339 P f20d775ac36948528aadf2fb37a3ed11: Neither blocks nor log segments found. Creating new log.
I20260926 03:44:00.580518 3137 tablet_bootstrap.cc:492] T e315dbe4cfeb4c23aaa910a6cbda3339 P f496bcbb6d8644ad962eceeeac3f82ca: Bootstrap starting.
I20260926 03:44:00.582055 3137 tablet_bootstrap.cc:654] T e315dbe4cfeb4c23aaa910a6cbda3339 P f496bcbb6d8644ad962eceeeac3f82ca: Neither blocks nor log segments found. Creating new log.
I20260926 03:44:00.583981 3137 tablet_bootstrap.cc:492] T e315dbe4cfeb4c23aaa910a6cbda3339 P f496bcbb6d8644ad962eceeeac3f82ca: No bootstrap required, opened a new log
I20260926 03:44:00.584002 3138 tablet_bootstrap.cc:492] T e315dbe4cfeb4c23aaa910a6cbda3339 P 737f7fce56574ef6a528cd2a48e8056d: No bootstrap required, opened a new log
I20260926 03:44:00.584139 3137 ts_tablet_manager.cc:1397] T e315dbe4cfeb4c23aaa910a6cbda3339 P f496bcbb6d8644ad962eceeeac3f82ca: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260926 03:44:00.584164 3138 ts_tablet_manager.cc:1397] T e315dbe4cfeb4c23aaa910a6cbda3339 P 737f7fce56574ef6a528cd2a48e8056d: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20260926 03:44:00.584738 3137 raft_consensus.cc:348] T e315dbe4cfeb4c23aaa910a6cbda3339 P f496bcbb6d8644ad962eceeeac3f82ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f496bcbb6d8644ad962eceeeac3f82ca" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 36187 } } peers { permanent_uuid: "f20d775ac36948528aadf2fb37a3ed11" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 42655 } } peers { permanent_uuid: "737f7fce56574ef6a528cd2a48e8056d" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 39703 } }
I20260926 03:44:00.584946 3137 raft_consensus.cc:374] T e315dbe4cfeb4c23aaa910a6cbda3339 P f496bcbb6d8644ad962eceeeac3f82ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:44:00.584815 3138 raft_consensus.cc:348] T e315dbe4cfeb4c23aaa910a6cbda3339 P 737f7fce56574ef6a528cd2a48e8056d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f496bcbb6d8644ad962eceeeac3f82ca" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 36187 } } peers { permanent_uuid: "f20d775ac36948528aadf2fb37a3ed11" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 42655 } } peers { permanent_uuid: "737f7fce56574ef6a528cd2a48e8056d" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 39703 } }
I20260926 03:44:00.585053 3138 raft_consensus.cc:374] T e315dbe4cfeb4c23aaa910a6cbda3339 P 737f7fce56574ef6a528cd2a48e8056d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:44:00.585064 3137 raft_consensus.cc:729] T e315dbe4cfeb4c23aaa910a6cbda3339 P f496bcbb6d8644ad962eceeeac3f82ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f496bcbb6d8644ad962eceeeac3f82ca, State: Initialized, Role: FOLLOWER
I20260926 03:44:00.585136 3138 raft_consensus.cc:729] T e315dbe4cfeb4c23aaa910a6cbda3339 P 737f7fce56574ef6a528cd2a48e8056d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 737f7fce56574ef6a528cd2a48e8056d, State: Initialized, Role: FOLLOWER
I20260926 03:44:00.585302 3137 consensus_queue.cc:260] T e315dbe4cfeb4c23aaa910a6cbda3339 P f496bcbb6d8644ad962eceeeac3f82ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f496bcbb6d8644ad962eceeeac3f82ca" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 36187 } } peers { permanent_uuid: "f20d775ac36948528aadf2fb37a3ed11" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 42655 } } peers { permanent_uuid: "737f7fce56574ef6a528cd2a48e8056d" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 39703 } }
I20260926 03:44:00.585386 3138 consensus_queue.cc:260] T e315dbe4cfeb4c23aaa910a6cbda3339 P 737f7fce56574ef6a528cd2a48e8056d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f496bcbb6d8644ad962eceeeac3f82ca" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 36187 } } peers { permanent_uuid: "f20d775ac36948528aadf2fb37a3ed11" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 42655 } } peers { permanent_uuid: "737f7fce56574ef6a528cd2a48e8056d" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 39703 } }
I20260926 03:44:00.585940 3138 ts_tablet_manager.cc:1428] T e315dbe4cfeb4c23aaa910a6cbda3339 P 737f7fce56574ef6a528cd2a48e8056d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 03:44:00.586014 3137 ts_tablet_manager.cc:1428] T e315dbe4cfeb4c23aaa910a6cbda3339 P f496bcbb6d8644ad962eceeeac3f82ca: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 03:44:00.587695 3136 tablet_bootstrap.cc:492] T e315dbe4cfeb4c23aaa910a6cbda3339 P f20d775ac36948528aadf2fb37a3ed11: No bootstrap required, opened a new log
I20260926 03:44:00.587865 3136 ts_tablet_manager.cc:1397] T e315dbe4cfeb4c23aaa910a6cbda3339 P f20d775ac36948528aadf2fb37a3ed11: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260926 03:44:00.588745 3136 raft_consensus.cc:348] T e315dbe4cfeb4c23aaa910a6cbda3339 P f20d775ac36948528aadf2fb37a3ed11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f496bcbb6d8644ad962eceeeac3f82ca" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 36187 } } peers { permanent_uuid: "f20d775ac36948528aadf2fb37a3ed11" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 42655 } } peers { permanent_uuid: "737f7fce56574ef6a528cd2a48e8056d" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 39703 } }
I20260926 03:44:00.589015 3136 raft_consensus.cc:374] T e315dbe4cfeb4c23aaa910a6cbda3339 P f20d775ac36948528aadf2fb37a3ed11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:44:00.589134 3136 raft_consensus.cc:729] T e315dbe4cfeb4c23aaa910a6cbda3339 P f20d775ac36948528aadf2fb37a3ed11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f20d775ac36948528aadf2fb37a3ed11, State: Initialized, Role: FOLLOWER
I20260926 03:44:00.589465 3136 consensus_queue.cc:260] T e315dbe4cfeb4c23aaa910a6cbda3339 P f20d775ac36948528aadf2fb37a3ed11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f496bcbb6d8644ad962eceeeac3f82ca" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 36187 } } peers { permanent_uuid: "f20d775ac36948528aadf2fb37a3ed11" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 42655 } } peers { permanent_uuid: "737f7fce56574ef6a528cd2a48e8056d" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 39703 } }
I20260926 03:44:00.590349 3136 ts_tablet_manager.cc:1428] T e315dbe4cfeb4c23aaa910a6cbda3339 P f20d775ac36948528aadf2fb37a3ed11: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260926 03:44:00.689975 3101 raft_consensus.cc:482] T e315dbe4cfeb4c23aaa910a6cbda3339 P 737f7fce56574ef6a528cd2a48e8056d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 03:44:00.690246 3101 raft_consensus.cc:504] T e315dbe4cfeb4c23aaa910a6cbda3339 P 737f7fce56574ef6a528cd2a48e8056d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f496bcbb6d8644ad962eceeeac3f82ca" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 36187 } } peers { permanent_uuid: "f20d775ac36948528aadf2fb37a3ed11" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 42655 } } peers { permanent_uuid: "737f7fce56574ef6a528cd2a48e8056d" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 39703 } }
I20260926 03:44:00.690953 3101 leader_election.cc:288] T e315dbe4cfeb4c23aaa910a6cbda3339 P 737f7fce56574ef6a528cd2a48e8056d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f496bcbb6d8644ad962eceeeac3f82ca (127.0.194.193:36187), f20d775ac36948528aadf2fb37a3ed11 (127.0.194.194:42655)
I20260926 03:44:00.691732 2687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e315dbe4cfeb4c23aaa910a6cbda3339" candidate_uuid: "737f7fce56574ef6a528cd2a48e8056d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f496bcbb6d8644ad962eceeeac3f82ca" is_pre_election: true
I20260926 03:44:00.691748 2827 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e315dbe4cfeb4c23aaa910a6cbda3339" candidate_uuid: "737f7fce56574ef6a528cd2a48e8056d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f20d775ac36948528aadf2fb37a3ed11" is_pre_election: true
I20260926 03:44:00.691999 2827 raft_consensus.cc:2454] T e315dbe4cfeb4c23aaa910a6cbda3339 P f20d775ac36948528aadf2fb37a3ed11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 737f7fce56574ef6a528cd2a48e8056d in term 0.
I20260926 03:44:00.692018 2687 raft_consensus.cc:2454] T e315dbe4cfeb4c23aaa910a6cbda3339 P f496bcbb6d8644ad962eceeeac3f82ca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 737f7fce56574ef6a528cd2a48e8056d in term 0.
I20260926 03:44:00.692500 2895 leader_election.cc:302] T e315dbe4cfeb4c23aaa910a6cbda3339 P 737f7fce56574ef6a528cd2a48e8056d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 737f7fce56574ef6a528cd2a48e8056d, f496bcbb6d8644ad962eceeeac3f82ca; no voters:
I20260926 03:44:00.692765 3101 raft_consensus.cc:2781] T e315dbe4cfeb4c23aaa910a6cbda3339 P 737f7fce56574ef6a528cd2a48e8056d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 03:44:00.692914 3101 raft_consensus.cc:482] T e315dbe4cfeb4c23aaa910a6cbda3339 P 737f7fce56574ef6a528cd2a48e8056d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 03:44:00.693005 3101 raft_consensus.cc:3037] T e315dbe4cfeb4c23aaa910a6cbda3339 P 737f7fce56574ef6a528cd2a48e8056d [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:44:00.695077 3101 raft_consensus.cc:504] T e315dbe4cfeb4c23aaa910a6cbda3339 P 737f7fce56574ef6a528cd2a48e8056d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f496bcbb6d8644ad962eceeeac3f82ca" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 36187 } } peers { permanent_uuid: "f20d775ac36948528aadf2fb37a3ed11" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 42655 } } peers { permanent_uuid: "737f7fce56574ef6a528cd2a48e8056d" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 39703 } }
I20260926 03:44:00.695631 3101 leader_election.cc:288] T e315dbe4cfeb4c23aaa910a6cbda3339 P 737f7fce56574ef6a528cd2a48e8056d [CANDIDATE]: Term 1 election: Requested vote from peers f496bcbb6d8644ad962eceeeac3f82ca (127.0.194.193:36187), f20d775ac36948528aadf2fb37a3ed11 (127.0.194.194:42655)
I20260926 03:44:00.696130 2687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e315dbe4cfeb4c23aaa910a6cbda3339" candidate_uuid: "737f7fce56574ef6a528cd2a48e8056d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f496bcbb6d8644ad962eceeeac3f82ca"
I20260926 03:44:00.696192 2827 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e315dbe4cfeb4c23aaa910a6cbda3339" candidate_uuid: "737f7fce56574ef6a528cd2a48e8056d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f20d775ac36948528aadf2fb37a3ed11"
I20260926 03:44:00.696355 2687 raft_consensus.cc:3037] T e315dbe4cfeb4c23aaa910a6cbda3339 P f496bcbb6d8644ad962eceeeac3f82ca [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:44:00.696475 2827 raft_consensus.cc:3037] T e315dbe4cfeb4c23aaa910a6cbda3339 P f20d775ac36948528aadf2fb37a3ed11 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:44:00.698873 2687 raft_consensus.cc:2454] T e315dbe4cfeb4c23aaa910a6cbda3339 P f496bcbb6d8644ad962eceeeac3f82ca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 737f7fce56574ef6a528cd2a48e8056d in term 1.
I20260926 03:44:00.698864 2827 raft_consensus.cc:2454] T e315dbe4cfeb4c23aaa910a6cbda3339 P f20d775ac36948528aadf2fb37a3ed11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 737f7fce56574ef6a528cd2a48e8056d in term 1.
I20260926 03:44:00.699290 2895 leader_election.cc:302] T e315dbe4cfeb4c23aaa910a6cbda3339 P 737f7fce56574ef6a528cd2a48e8056d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 737f7fce56574ef6a528cd2a48e8056d, f496bcbb6d8644ad962eceeeac3f82ca; no voters:
I20260926 03:44:00.699618 3101 raft_consensus.cc:2781] T e315dbe4cfeb4c23aaa910a6cbda3339 P 737f7fce56574ef6a528cd2a48e8056d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:44:00.699785 3101 raft_consensus.cc:686] T e315dbe4cfeb4c23aaa910a6cbda3339 P 737f7fce56574ef6a528cd2a48e8056d [term 1 LEADER]: Becoming Leader. State: Replica: 737f7fce56574ef6a528cd2a48e8056d, State: Running, Role: LEADER
I20260926 03:44:00.700102 3101 consensus_queue.cc:237] T e315dbe4cfeb4c23aaa910a6cbda3339 P 737f7fce56574ef6a528cd2a48e8056d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f496bcbb6d8644ad962eceeeac3f82ca" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 36187 } } peers { permanent_uuid: "f20d775ac36948528aadf2fb37a3ed11" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 42655 } } peers { permanent_uuid: "737f7fce56574ef6a528cd2a48e8056d" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 39703 } }
I20260926 03:44:00.702621 2526 catalog_manager.cc:5495] T e315dbe4cfeb4c23aaa910a6cbda3339 P 737f7fce56574ef6a528cd2a48e8056d reported cstate change: term changed from 0 to 1, leader changed from <none> to 737f7fce56574ef6a528cd2a48e8056d (127.0.194.195). New cstate: current_term: 1 leader_uuid: "737f7fce56574ef6a528cd2a48e8056d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f496bcbb6d8644ad962eceeeac3f82ca" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 36187 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f20d775ac36948528aadf2fb37a3ed11" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 42655 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "737f7fce56574ef6a528cd2a48e8056d" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 39703 } health_report { overall_health: HEALTHY } } }
I20260926 03:44:00.747044 3145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:00.741890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60102 (local address 127.0.194.195:39703)
0926 03:44:00.742534 (+ 644us) server_negotiation.cc:206] Beginning negotiation
0926 03:44:00.742538 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:44:00.742557 (+ 19us) server_negotiation.cc:407] Connection header received
0926 03:44:00.742611 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:44:00.742614 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:44:00.742672 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:44:00.742808 (+ 136us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:44:00.743232 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:00.744135 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:00.745407 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:00.745820 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:00.745913 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:00.746281 (+ 368us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:44:00.746378 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:44:00.746445 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 03:44:00.746587 (+ 142us) server_negotiation.cc:299] Negotiation successful
0926 03:44:00.746646 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":534,"thread_start_us":411,"threads_started":1}
I20260926 03:44:00.755733 2827 raft_consensus.cc:1261] T e315dbe4cfeb4c23aaa910a6cbda3339 P f20d775ac36948528aadf2fb37a3ed11 [term 1 FOLLOWER]: Refusing update from remote peer 737f7fce56574ef6a528cd2a48e8056d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:44:00.755929 2687 raft_consensus.cc:1261] T e315dbe4cfeb4c23aaa910a6cbda3339 P f496bcbb6d8644ad962eceeeac3f82ca [term 1 FOLLOWER]: Refusing update from remote peer 737f7fce56574ef6a528cd2a48e8056d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:44:00.756376 3120 consensus_queue.cc:1035] T e315dbe4cfeb4c23aaa910a6cbda3339 P 737f7fce56574ef6a528cd2a48e8056d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f20d775ac36948528aadf2fb37a3ed11" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 42655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:44:00.756685 3101 consensus_queue.cc:1035] T e315dbe4cfeb4c23aaa910a6cbda3339 P 737f7fce56574ef6a528cd2a48e8056d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f496bcbb6d8644ad962eceeeac3f82ca" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 36187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:44:00.801329 2526 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56064:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:44:00.801584 2526 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56064:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:44:00.804701 2526 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0ccb5168dad34eb2a79efa733da1e3f6: Sending DeleteTablet for 3 replicas of tablet e315dbe4cfeb4c23aaa910a6cbda3339
I20260926 03:44:00.805648 2940 tablet_service.cc:1514] Processing DeleteTablet for tablet e315dbe4cfeb4c23aaa910a6cbda3339 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:44:00 UTC) from {username='kudu', principal='kudu/127.0.194.254@KRBTEST.COM'} at 127.0.0.1:60094
I20260926 03:44:00.805785 2663 tablet_service.cc:1514] Processing DeleteTablet for tablet e315dbe4cfeb4c23aaa910a6cbda3339 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:44:00 UTC) from {username='kudu', principal='kudu/127.0.194.254@KRBTEST.COM'} at 127.0.0.1:39770
I20260926 03:44:00.805930 3127 tablet_replica.cc:330] T e315dbe4cfeb4c23aaa910a6cbda3339 P 737f7fce56574ef6a528cd2a48e8056d: stopping tablet replica
I20260926 03:44:00.806085 2802 tablet_service.cc:1514] Processing DeleteTablet for tablet e315dbe4cfeb4c23aaa910a6cbda3339 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:44:00 UTC) from {username='kudu', principal='kudu/127.0.194.254@KRBTEST.COM'} at 127.0.0.1:44504
I20260926 03:44:00.806133 3127 raft_consensus.cc:2229] T e315dbe4cfeb4c23aaa910a6cbda3339 P 737f7fce56574ef6a528cd2a48e8056d [term 1 LEADER]: Raft consensus shutting down.
I20260926 03:44:00.806319 3126 tablet_replica.cc:330] T e315dbe4cfeb4c23aaa910a6cbda3339 P f20d775ac36948528aadf2fb37a3ed11: stopping tablet replica
I20260926 03:44:00.806365 3125 tablet_replica.cc:330] T e315dbe4cfeb4c23aaa910a6cbda3339 P f496bcbb6d8644ad962eceeeac3f82ca: stopping tablet replica
I20260926 03:44:00.806468 3127 raft_consensus.cc:2258] T e315dbe4cfeb4c23aaa910a6cbda3339 P 737f7fce56574ef6a528cd2a48e8056d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:44:00.806497 3126 raft_consensus.cc:2229] T e315dbe4cfeb4c23aaa910a6cbda3339 P f20d775ac36948528aadf2fb37a3ed11 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:44:00.806545 3125 raft_consensus.cc:2229] T e315dbe4cfeb4c23aaa910a6cbda3339 P f496bcbb6d8644ad962eceeeac3f82ca [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:44:00.806680 3126 raft_consensus.cc:2258] T e315dbe4cfeb4c23aaa910a6cbda3339 P f20d775ac36948528aadf2fb37a3ed11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:44:00.806703 3125 raft_consensus.cc:2258] T e315dbe4cfeb4c23aaa910a6cbda3339 P f496bcbb6d8644ad962eceeeac3f82ca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:44:00.806669 2526 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56064:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 03:44:00.807380 2526 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 03:44:00.807682 3127 ts_tablet_manager.cc:1905] T e315dbe4cfeb4c23aaa910a6cbda3339 P 737f7fce56574ef6a528cd2a48e8056d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:44:00.807781 3126 ts_tablet_manager.cc:1905] T e315dbe4cfeb4c23aaa910a6cbda3339 P f20d775ac36948528aadf2fb37a3ed11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:44:00.814524 3125 ts_tablet_manager.cc:1905] T e315dbe4cfeb4c23aaa910a6cbda3339 P f496bcbb6d8644ad962eceeeac3f82ca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:44:00.814710 3127 ts_tablet_manager.cc:1918] T e315dbe4cfeb4c23aaa910a6cbda3339 P 737f7fce56574ef6a528cd2a48e8056d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 03:44:00.814889 3127 log.cc:1196] T e315dbe4cfeb4c23aaa910a6cbda3339 P 737f7fce56574ef6a528cd2a48e8056d: Deleting WAL directory at /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790394221564982-779-0/minicluster-data/ts-2/wal/wals/e315dbe4cfeb4c23aaa910a6cbda3339
I20260926 03:44:00.815248 3127 ts_tablet_manager.cc:1939] T e315dbe4cfeb4c23aaa910a6cbda3339 P 737f7fce56574ef6a528cd2a48e8056d: Deleting consensus metadata
I20260926 03:44:00.816318 2510 catalog_manager.cc:4841] TS 737f7fce56574ef6a528cd2a48e8056d (127.0.194.195:39703): tablet e315dbe4cfeb4c23aaa910a6cbda3339 (table test-table [id=5084c26562b545ba9b07c5994becb632]) successfully deleted
I20260926 03:44:00.821429 3126 ts_tablet_manager.cc:1918] T e315dbe4cfeb4c23aaa910a6cbda3339 P f20d775ac36948528aadf2fb37a3ed11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 03:44:00.821621 3126 log.cc:1196] T e315dbe4cfeb4c23aaa910a6cbda3339 P f20d775ac36948528aadf2fb37a3ed11: Deleting WAL directory at /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790394221564982-779-0/minicluster-data/ts-1/wal/wals/e315dbe4cfeb4c23aaa910a6cbda3339
I20260926 03:44:00.821583 2802 tablet_service.cc:1467] Processing CreateTablet for tablet ff199c5c4b254df99ec8a8414c2a0e31 (DEFAULT_TABLE table=test-table [id=b3b6d3dd9d054fb9827400828d1dbf92]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:44:00.821728 2940 tablet_service.cc:1467] Processing CreateTablet for tablet ff199c5c4b254df99ec8a8414c2a0e31 (DEFAULT_TABLE table=test-table [id=b3b6d3dd9d054fb9827400828d1dbf92]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:44:00.822046 2802 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff199c5c4b254df99ec8a8414c2a0e31. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:44:00.822126 2940 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff199c5c4b254df99ec8a8414c2a0e31. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:44:00.822072 3126 ts_tablet_manager.cc:1939] T e315dbe4cfeb4c23aaa910a6cbda3339 P f20d775ac36948528aadf2fb37a3ed11: Deleting consensus metadata
I20260926 03:44:00.823562 2513 catalog_manager.cc:4841] TS f20d775ac36948528aadf2fb37a3ed11 (127.0.194.194:42655): tablet e315dbe4cfeb4c23aaa910a6cbda3339 (table test-table [id=5084c26562b545ba9b07c5994becb632]) successfully deleted
I20260926 03:44:00.824365 2663 tablet_service.cc:1467] Processing CreateTablet for tablet ff199c5c4b254df99ec8a8414c2a0e31 (DEFAULT_TABLE table=test-table [id=b3b6d3dd9d054fb9827400828d1dbf92]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:44:00.824791 2663 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff199c5c4b254df99ec8a8414c2a0e31. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:44:00.828380 3136 tablet_bootstrap.cc:492] T ff199c5c4b254df99ec8a8414c2a0e31 P f20d775ac36948528aadf2fb37a3ed11: Bootstrap starting.
I20260926 03:44:00.828397 3138 tablet_bootstrap.cc:492] T ff199c5c4b254df99ec8a8414c2a0e31 P 737f7fce56574ef6a528cd2a48e8056d: Bootstrap starting.
I20260926 03:44:00.829975 3125 ts_tablet_manager.cc:1918] T e315dbe4cfeb4c23aaa910a6cbda3339 P f496bcbb6d8644ad962eceeeac3f82ca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 03:44:00.830029 3138 tablet_bootstrap.cc:654] T ff199c5c4b254df99ec8a8414c2a0e31 P 737f7fce56574ef6a528cd2a48e8056d: Neither blocks nor log segments found. Creating new log.
I20260926 03:44:00.830068 3136 tablet_bootstrap.cc:654] T ff199c5c4b254df99ec8a8414c2a0e31 P f20d775ac36948528aadf2fb37a3ed11: Neither blocks nor log segments found. Creating new log.
I20260926 03:44:00.830323 3125 log.cc:1196] T e315dbe4cfeb4c23aaa910a6cbda3339 P f496bcbb6d8644ad962eceeeac3f82ca: Deleting WAL directory at /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790394221564982-779-0/minicluster-data/ts-0/wal/wals/e315dbe4cfeb4c23aaa910a6cbda3339
I20260926 03:44:00.830899 3125 ts_tablet_manager.cc:1939] T e315dbe4cfeb4c23aaa910a6cbda3339 P f496bcbb6d8644ad962eceeeac3f82ca: Deleting consensus metadata
I20260926 03:44:00.831523 3137 tablet_bootstrap.cc:492] T ff199c5c4b254df99ec8a8414c2a0e31 P f496bcbb6d8644ad962eceeeac3f82ca: Bootstrap starting.
I20260926 03:44:00.832489 2513 catalog_manager.cc:4841] TS f496bcbb6d8644ad962eceeeac3f82ca (127.0.194.193:36187): tablet e315dbe4cfeb4c23aaa910a6cbda3339 (table test-table [id=5084c26562b545ba9b07c5994becb632]) successfully deleted
I20260926 03:44:00.833391 3137 tablet_bootstrap.cc:654] T ff199c5c4b254df99ec8a8414c2a0e31 P f496bcbb6d8644ad962eceeeac3f82ca: Neither blocks nor log segments found. Creating new log.
I20260926 03:44:00.843351 3136 tablet_bootstrap.cc:492] T ff199c5c4b254df99ec8a8414c2a0e31 P f20d775ac36948528aadf2fb37a3ed11: No bootstrap required, opened a new log
I20260926 03:44:00.843896 3136 ts_tablet_manager.cc:1397] T ff199c5c4b254df99ec8a8414c2a0e31 P f20d775ac36948528aadf2fb37a3ed11: Time spent bootstrapping tablet: real 0.015s user 0.009s sys 0.006s
I20260926 03:44:00.844250 3138 tablet_bootstrap.cc:492] T ff199c5c4b254df99ec8a8414c2a0e31 P 737f7fce56574ef6a528cd2a48e8056d: No bootstrap required, opened a new log
I20260926 03:44:00.844466 3138 ts_tablet_manager.cc:1397] T ff199c5c4b254df99ec8a8414c2a0e31 P 737f7fce56574ef6a528cd2a48e8056d: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.007s
I20260926 03:44:00.844792 3136 raft_consensus.cc:348] T ff199c5c4b254df99ec8a8414c2a0e31 P f20d775ac36948528aadf2fb37a3ed11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f496bcbb6d8644ad962eceeeac3f82ca" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 36187 } } peers { permanent_uuid: "f20d775ac36948528aadf2fb37a3ed11" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 42655 } } peers { permanent_uuid: "737f7fce56574ef6a528cd2a48e8056d" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 39703 } }
I20260926 03:44:00.845111 3136 raft_consensus.cc:374] T ff199c5c4b254df99ec8a8414c2a0e31 P f20d775ac36948528aadf2fb37a3ed11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:44:00.845237 3136 raft_consensus.cc:729] T ff199c5c4b254df99ec8a8414c2a0e31 P f20d775ac36948528aadf2fb37a3ed11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f20d775ac36948528aadf2fb37a3ed11, State: Initialized, Role: FOLLOWER
I20260926 03:44:00.845211 3138 raft_consensus.cc:348] T ff199c5c4b254df99ec8a8414c2a0e31 P 737f7fce56574ef6a528cd2a48e8056d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f496bcbb6d8644ad962eceeeac3f82ca" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 36187 } } peers { permanent_uuid: "f20d775ac36948528aadf2fb37a3ed11" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 42655 } } peers { permanent_uuid: "737f7fce56574ef6a528cd2a48e8056d" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 39703 } }
I20260926 03:44:00.845520 3138 raft_consensus.cc:374] T ff199c5c4b254df99ec8a8414c2a0e31 P 737f7fce56574ef6a528cd2a48e8056d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:44:00.845628 3138 raft_consensus.cc:729] T ff199c5c4b254df99ec8a8414c2a0e31 P 737f7fce56574ef6a528cd2a48e8056d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 737f7fce56574ef6a528cd2a48e8056d, State: Initialized, Role: FOLLOWER
I20260926 03:44:00.845536 3136 consensus_queue.cc:260] T ff199c5c4b254df99ec8a8414c2a0e31 P f20d775ac36948528aadf2fb37a3ed11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f496bcbb6d8644ad962eceeeac3f82ca" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 36187 } } peers { permanent_uuid: "f20d775ac36948528aadf2fb37a3ed11" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 42655 } } peers { permanent_uuid: "737f7fce56574ef6a528cd2a48e8056d" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 39703 } }
I20260926 03:44:00.845880 3138 consensus_queue.cc:260] T ff199c5c4b254df99ec8a8414c2a0e31 P 737f7fce56574ef6a528cd2a48e8056d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f496bcbb6d8644ad962eceeeac3f82ca" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 36187 } } peers { permanent_uuid: "f20d775ac36948528aadf2fb37a3ed11" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 42655 } } peers { permanent_uuid: "737f7fce56574ef6a528cd2a48e8056d" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 39703 } }
I20260926 03:44:00.846446 3138 ts_tablet_manager.cc:1428] T ff199c5c4b254df99ec8a8414c2a0e31 P 737f7fce56574ef6a528cd2a48e8056d: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20260926 03:44:00.847012 3136 ts_tablet_manager.cc:1428] T ff199c5c4b254df99ec8a8414c2a0e31 P f20d775ac36948528aadf2fb37a3ed11: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260926 03:44:00.847565 3137 tablet_bootstrap.cc:492] T ff199c5c4b254df99ec8a8414c2a0e31 P f496bcbb6d8644ad962eceeeac3f82ca: No bootstrap required, opened a new log
I20260926 03:44:00.847741 3137 ts_tablet_manager.cc:1397] T ff199c5c4b254df99ec8a8414c2a0e31 P f496bcbb6d8644ad962eceeeac3f82ca: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.005s
I20260926 03:44:00.848524 3137 raft_consensus.cc:348] T ff199c5c4b254df99ec8a8414c2a0e31 P f496bcbb6d8644ad962eceeeac3f82ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f496bcbb6d8644ad962eceeeac3f82ca" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 36187 } } peers { permanent_uuid: "f20d775ac36948528aadf2fb37a3ed11" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 42655 } } peers { permanent_uuid: "737f7fce56574ef6a528cd2a48e8056d" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 39703 } }
I20260926 03:44:00.848793 3137 raft_consensus.cc:374] T ff199c5c4b254df99ec8a8414c2a0e31 P f496bcbb6d8644ad962eceeeac3f82ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:44:00.848883 3137 raft_consensus.cc:729] T ff199c5c4b254df99ec8a8414c2a0e31 P f496bcbb6d8644ad962eceeeac3f82ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f496bcbb6d8644ad962eceeeac3f82ca, State: Initialized, Role: FOLLOWER
I20260926 03:44:00.849138 3137 consensus_queue.cc:260] T ff199c5c4b254df99ec8a8414c2a0e31 P f496bcbb6d8644ad962eceeeac3f82ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f496bcbb6d8644ad962eceeeac3f82ca" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 36187 } } peers { permanent_uuid: "f20d775ac36948528aadf2fb37a3ed11" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 42655 } } peers { permanent_uuid: "737f7fce56574ef6a528cd2a48e8056d" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 39703 } }
I20260926 03:44:00.849714 3137 ts_tablet_manager.cc:1428] T ff199c5c4b254df99ec8a8414c2a0e31 P f496bcbb6d8644ad962eceeeac3f82ca: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260926 03:44:00.957463 3101 raft_consensus.cc:482] T ff199c5c4b254df99ec8a8414c2a0e31 P 737f7fce56574ef6a528cd2a48e8056d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 03:44:00.957691 3101 raft_consensus.cc:504] T ff199c5c4b254df99ec8a8414c2a0e31 P 737f7fce56574ef6a528cd2a48e8056d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f496bcbb6d8644ad962eceeeac3f82ca" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 36187 } } peers { permanent_uuid: "f20d775ac36948528aadf2fb37a3ed11" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 42655 } } peers { permanent_uuid: "737f7fce56574ef6a528cd2a48e8056d" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 39703 } }
I20260926 03:44:00.958283 3101 leader_election.cc:288] T ff199c5c4b254df99ec8a8414c2a0e31 P 737f7fce56574ef6a528cd2a48e8056d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f496bcbb6d8644ad962eceeeac3f82ca (127.0.194.193:36187), f20d775ac36948528aadf2fb37a3ed11 (127.0.194.194:42655)
I20260926 03:44:00.958886 2687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff199c5c4b254df99ec8a8414c2a0e31" candidate_uuid: "737f7fce56574ef6a528cd2a48e8056d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f496bcbb6d8644ad962eceeeac3f82ca" is_pre_election: true
I20260926 03:44:00.958956 2827 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff199c5c4b254df99ec8a8414c2a0e31" candidate_uuid: "737f7fce56574ef6a528cd2a48e8056d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f20d775ac36948528aadf2fb37a3ed11" is_pre_election: true
I20260926 03:44:00.959184 2687 raft_consensus.cc:2454] T ff199c5c4b254df99ec8a8414c2a0e31 P f496bcbb6d8644ad962eceeeac3f82ca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 737f7fce56574ef6a528cd2a48e8056d in term 0.
I20260926 03:44:00.959229 2827 raft_consensus.cc:2454] T ff199c5c4b254df99ec8a8414c2a0e31 P f20d775ac36948528aadf2fb37a3ed11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 737f7fce56574ef6a528cd2a48e8056d in term 0.
I20260926 03:44:00.959641 2895 leader_election.cc:302] T ff199c5c4b254df99ec8a8414c2a0e31 P 737f7fce56574ef6a528cd2a48e8056d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 737f7fce56574ef6a528cd2a48e8056d, f496bcbb6d8644ad962eceeeac3f82ca; no voters:
I20260926 03:44:00.959964 3101 raft_consensus.cc:2781] T ff199c5c4b254df99ec8a8414c2a0e31 P 737f7fce56574ef6a528cd2a48e8056d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 03:44:00.960101 3101 raft_consensus.cc:482] T ff199c5c4b254df99ec8a8414c2a0e31 P 737f7fce56574ef6a528cd2a48e8056d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 03:44:00.960207 3101 raft_consensus.cc:3037] T ff199c5c4b254df99ec8a8414c2a0e31 P 737f7fce56574ef6a528cd2a48e8056d [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:44:00.962455 3101 raft_consensus.cc:504] T ff199c5c4b254df99ec8a8414c2a0e31 P 737f7fce56574ef6a528cd2a48e8056d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f496bcbb6d8644ad962eceeeac3f82ca" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 36187 } } peers { permanent_uuid: "f20d775ac36948528aadf2fb37a3ed11" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 42655 } } peers { permanent_uuid: "737f7fce56574ef6a528cd2a48e8056d" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 39703 } }
I20260926 03:44:00.963030 3101 leader_election.cc:288] T ff199c5c4b254df99ec8a8414c2a0e31 P 737f7fce56574ef6a528cd2a48e8056d [CANDIDATE]: Term 1 election: Requested vote from peers f496bcbb6d8644ad962eceeeac3f82ca (127.0.194.193:36187), f20d775ac36948528aadf2fb37a3ed11 (127.0.194.194:42655)
I20260926 03:44:00.963725 2827 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff199c5c4b254df99ec8a8414c2a0e31" candidate_uuid: "737f7fce56574ef6a528cd2a48e8056d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f20d775ac36948528aadf2fb37a3ed11"
I20260926 03:44:00.963707 2687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff199c5c4b254df99ec8a8414c2a0e31" candidate_uuid: "737f7fce56574ef6a528cd2a48e8056d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f496bcbb6d8644ad962eceeeac3f82ca"
I20260926 03:44:00.963953 2827 raft_consensus.cc:3037] T ff199c5c4b254df99ec8a8414c2a0e31 P f20d775ac36948528aadf2fb37a3ed11 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:44:00.963991 2687 raft_consensus.cc:3037] T ff199c5c4b254df99ec8a8414c2a0e31 P f496bcbb6d8644ad962eceeeac3f82ca [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:44:00.965960 2827 raft_consensus.cc:2454] T ff199c5c4b254df99ec8a8414c2a0e31 P f20d775ac36948528aadf2fb37a3ed11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 737f7fce56574ef6a528cd2a48e8056d in term 1.
I20260926 03:44:00.965945 2687 raft_consensus.cc:2454] T ff199c5c4b254df99ec8a8414c2a0e31 P f496bcbb6d8644ad962eceeeac3f82ca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 737f7fce56574ef6a528cd2a48e8056d in term 1.
I20260926 03:44:00.966531 2895 leader_election.cc:302] T ff199c5c4b254df99ec8a8414c2a0e31 P 737f7fce56574ef6a528cd2a48e8056d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 737f7fce56574ef6a528cd2a48e8056d, f20d775ac36948528aadf2fb37a3ed11; no voters:
I20260926 03:44:00.966912 3101 raft_consensus.cc:2781] T ff199c5c4b254df99ec8a8414c2a0e31 P 737f7fce56574ef6a528cd2a48e8056d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:44:00.967085 3101 raft_consensus.cc:686] T ff199c5c4b254df99ec8a8414c2a0e31 P 737f7fce56574ef6a528cd2a48e8056d [term 1 LEADER]: Becoming Leader. State: Replica: 737f7fce56574ef6a528cd2a48e8056d, State: Running, Role: LEADER
I20260926 03:44:00.967357 3101 consensus_queue.cc:237] T ff199c5c4b254df99ec8a8414c2a0e31 P 737f7fce56574ef6a528cd2a48e8056d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f496bcbb6d8644ad962eceeeac3f82ca" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 36187 } } peers { permanent_uuid: "f20d775ac36948528aadf2fb37a3ed11" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 42655 } } peers { permanent_uuid: "737f7fce56574ef6a528cd2a48e8056d" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 39703 } }
I20260926 03:44:00.970232 2526 catalog_manager.cc:5495] T ff199c5c4b254df99ec8a8414c2a0e31 P 737f7fce56574ef6a528cd2a48e8056d reported cstate change: term changed from 0 to 1, leader changed from <none> to 737f7fce56574ef6a528cd2a48e8056d (127.0.194.195). New cstate: current_term: 1 leader_uuid: "737f7fce56574ef6a528cd2a48e8056d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f496bcbb6d8644ad962eceeeac3f82ca" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 36187 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f20d775ac36948528aadf2fb37a3ed11" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 42655 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "737f7fce56574ef6a528cd2a48e8056d" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 39703 } health_report { overall_health: HEALTHY } } }
I20260926 03:44:01.005339 2687 raft_consensus.cc:1261] T ff199c5c4b254df99ec8a8414c2a0e31 P f496bcbb6d8644ad962eceeeac3f82ca [term 1 FOLLOWER]: Refusing update from remote peer 737f7fce56574ef6a528cd2a48e8056d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:44:01.005432 2827 raft_consensus.cc:1261] T ff199c5c4b254df99ec8a8414c2a0e31 P f20d775ac36948528aadf2fb37a3ed11 [term 1 FOLLOWER]: Refusing update from remote peer 737f7fce56574ef6a528cd2a48e8056d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:44:01.005921 3120 consensus_queue.cc:1035] T ff199c5c4b254df99ec8a8414c2a0e31 P 737f7fce56574ef6a528cd2a48e8056d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f20d775ac36948528aadf2fb37a3ed11" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 42655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:44:01.006172 3101 consensus_queue.cc:1035] T ff199c5c4b254df99ec8a8414c2a0e31 P 737f7fce56574ef6a528cd2a48e8056d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f496bcbb6d8644ad962eceeeac3f82ca" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 36187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 03:44:01.026993 3107 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260926 03:44:02.631676 2524 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56064:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:44:02.631927 2524 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56064:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:44:02.635740 2524 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0ccb5168dad34eb2a79efa733da1e3f6: Sending DeleteTablet for 3 replicas of tablet ff199c5c4b254df99ec8a8414c2a0e31
I20260926 03:44:02.636533 2663 tablet_service.cc:1514] Processing DeleteTablet for tablet ff199c5c4b254df99ec8a8414c2a0e31 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:44:02 UTC) from {username='kudu', principal='kudu/127.0.194.254@KRBTEST.COM'} at 127.0.0.1:39770
I20260926 03:44:02.636600 2940 tablet_service.cc:1514] Processing DeleteTablet for tablet ff199c5c4b254df99ec8a8414c2a0e31 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:44:02 UTC) from {username='kudu', principal='kudu/127.0.194.254@KRBTEST.COM'} at 127.0.0.1:60094
I20260926 03:44:02.636623 2802 tablet_service.cc:1514] Processing DeleteTablet for tablet ff199c5c4b254df99ec8a8414c2a0e31 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:44:02 UTC) from {username='kudu', principal='kudu/127.0.194.254@KRBTEST.COM'} at 127.0.0.1:44504
I20260926 03:44:02.637326 779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu pid 2597
I20260926 03:44:02.652104 3181 tablet_replica.cc:330] T ff199c5c4b254df99ec8a8414c2a0e31 P f20d775ac36948528aadf2fb37a3ed11: stopping tablet replica
I20260926 03:44:02.652098 3180 tablet_replica.cc:330] T ff199c5c4b254df99ec8a8414c2a0e31 P 737f7fce56574ef6a528cd2a48e8056d: stopping tablet replica
I20260926 03:44:02.652369 3180 raft_consensus.cc:2229] T ff199c5c4b254df99ec8a8414c2a0e31 P 737f7fce56574ef6a528cd2a48e8056d [term 1 LEADER]: Raft consensus shutting down.
I20260926 03:44:02.652347 3181 raft_consensus.cc:2229] T ff199c5c4b254df99ec8a8414c2a0e31 P f20d775ac36948528aadf2fb37a3ed11 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:44:02.652663 3181 raft_consensus.cc:2258] T ff199c5c4b254df99ec8a8414c2a0e31 P f20d775ac36948528aadf2fb37a3ed11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:44:02.652822 3180 raft_consensus.cc:2258] T ff199c5c4b254df99ec8a8414c2a0e31 P 737f7fce56574ef6a528cd2a48e8056d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:44:02.654659 3180 ts_tablet_manager.cc:1905] T ff199c5c4b254df99ec8a8414c2a0e31 P 737f7fce56574ef6a528cd2a48e8056d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:44:02.654659 3181 ts_tablet_manager.cc:1905] T ff199c5c4b254df99ec8a8414c2a0e31 P f20d775ac36948528aadf2fb37a3ed11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:44:02.660933 3179 tablet_replica.cc:330] T ff199c5c4b254df99ec8a8414c2a0e31 P f496bcbb6d8644ad962eceeeac3f82ca: stopping tablet replica
I20260926 03:44:02.661175 3179 raft_consensus.cc:2229] T ff199c5c4b254df99ec8a8414c2a0e31 P f496bcbb6d8644ad962eceeeac3f82ca [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:44:02.661444 3179 raft_consensus.cc:2258] T ff199c5c4b254df99ec8a8414c2a0e31 P f496bcbb6d8644ad962eceeeac3f82ca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:44:02.662494 3179 ts_tablet_manager.cc:1905] T ff199c5c4b254df99ec8a8414c2a0e31 P f496bcbb6d8644ad962eceeeac3f82ca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:44:02.662591 3180 ts_tablet_manager.cc:1918] T ff199c5c4b254df99ec8a8414c2a0e31 P 737f7fce56574ef6a528cd2a48e8056d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 03:44:02.662724 3180 log.cc:1196] T ff199c5c4b254df99ec8a8414c2a0e31 P 737f7fce56574ef6a528cd2a48e8056d: Deleting WAL directory at /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790394221564982-779-0/minicluster-data/ts-2/wal/wals/ff199c5c4b254df99ec8a8414c2a0e31
I20260926 03:44:02.663208 3180 ts_tablet_manager.cc:1939] T ff199c5c4b254df99ec8a8414c2a0e31 P 737f7fce56574ef6a528cd2a48e8056d: Deleting consensus metadata
I20260926 03:44:02.663280 3181 ts_tablet_manager.cc:1918] T ff199c5c4b254df99ec8a8414c2a0e31 P f20d775ac36948528aadf2fb37a3ed11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 03:44:02.663441 3181 log.cc:1196] T ff199c5c4b254df99ec8a8414c2a0e31 P f20d775ac36948528aadf2fb37a3ed11: Deleting WAL directory at /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790394221564982-779-0/minicluster-data/ts-1/wal/wals/ff199c5c4b254df99ec8a8414c2a0e31
I20260926 03:44:02.663838 3181 ts_tablet_manager.cc:1939] T ff199c5c4b254df99ec8a8414c2a0e31 P f20d775ac36948528aadf2fb37a3ed11: Deleting consensus metadata
I20260926 03:44:02.664422 2510 catalog_manager.cc:4841] TS 737f7fce56574ef6a528cd2a48e8056d (127.0.194.195:39703): tablet ff199c5c4b254df99ec8a8414c2a0e31 (table test-table [id=b3b6d3dd9d054fb9827400828d1dbf92]) successfully deleted
I20260926 03:44:02.664834 2513 catalog_manager.cc:4841] TS f20d775ac36948528aadf2fb37a3ed11 (127.0.194.194:42655): tablet ff199c5c4b254df99ec8a8414c2a0e31 (table test-table [id=b3b6d3dd9d054fb9827400828d1dbf92]) successfully deleted
Sep 26 03:44:02 dist-test-slave-n2th krb5kdc[2481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394240, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.194.193@KRBTEST.COM
I20260926 03:44:02.668923 3179 ts_tablet_manager.cc:1918] T ff199c5c4b254df99ec8a8414c2a0e31 P f496bcbb6d8644ad962eceeeac3f82ca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 03:44:02.669093 3179 log.cc:1196] T ff199c5c4b254df99ec8a8414c2a0e31 P f496bcbb6d8644ad962eceeeac3f82ca: Deleting WAL directory at /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790394221564982-779-0/minicluster-data/ts-0/wal/wals/ff199c5c4b254df99ec8a8414c2a0e31
I20260926 03:44:02.669507 3179 ts_tablet_manager.cc:1939] T ff199c5c4b254df99ec8a8414c2a0e31 P f496bcbb6d8644ad962eceeeac3f82ca: Deleting consensus metadata
I20260926 03:44:02.670714 2513 catalog_manager.cc:4841] TS f496bcbb6d8644ad962eceeeac3f82ca (127.0.194.193:36187): tablet ff199c5c4b254df99ec8a8414c2a0e31 (table test-table [id=b3b6d3dd9d054fb9827400828d1dbf92]) successfully deleted
I20260926 03:44:02.675658 3182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:02.639002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39800 (local address 127.0.194.193:36187)
0926 03:44:02.653599 (+ 14597us) server_negotiation.cc:206] Beginning negotiation
0926 03:44:02.653606 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:44:02.654347 (+ 741us) server_negotiation.cc:407] Connection header received
0926 03:44:02.654420 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:44:02.654425 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:44:02.654505 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:44:02.654622 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:44:02.656546 (+ 1924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:02.657695 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:02.659511 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:02.660184 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:02.660310 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:02.669705 (+ 9395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:44:02.669729 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:44:02.669752 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:44:02.669792 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:44:02.672922 (+ 3130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:02.673596 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:02.673602 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:02.673605 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:02.673728 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:02.674095 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:02.674098 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:02.674100 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:02.674537 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:44:02.674813 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0926 03:44:02.675120 (+ 307us) server_negotiation.cc:299] Negotiation successful
0926 03:44:02.675392 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14414,"thread_start_us":14537,"threads_started":1}
W20260926 03:44:02.676314 2728 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39800
W20260926 03:44:02.676935 779 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu pid 2597: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 03:44:02.677161 779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu with pid 2597
I20260926 03:44:02.699752 779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu pid 2737
Sep 26 03:44:02 dist-test-slave-n2th krb5kdc[2481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394240, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.194.194@KRBTEST.COM
I20260926 03:44:02.715075 3184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:02.700550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44524 (local address 127.0.194.194:42655)
0926 03:44:02.701136 (+ 586us) server_negotiation.cc:206] Beginning negotiation
0926 03:44:02.701140 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:44:02.701156 (+ 16us) server_negotiation.cc:407] Connection header received
0926 03:44:02.701210 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:44:02.701213 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:44:02.701270 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:44:02.701363 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:44:02.702588 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:02.703447 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:02.704618 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:02.704983 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:02.705075 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:02.709323 (+ 4248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:44:02.709336 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:44:02.709353 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:44:02.709391 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:44:02.712334 (+ 2943us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:02.712975 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:02.712983 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:02.712985 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:02.713140 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:02.713442 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:02.713445 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:02.713447 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:02.713794 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:44:02.714022 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0926 03:44:02.714391 (+ 369us) server_negotiation.cc:299] Negotiation successful
0926 03:44:02.714635 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":473,"thread_start_us":333,"threads_started":1}
W20260926 03:44:02.715761 2867 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44524
W20260926 03:44:02.716364 779 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu pid 2737: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 03:44:02.716526 779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu with pid 2737
I20260926 03:44:02.739400 779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu pid 2876
Sep 26 03:44:02 dist-test-slave-n2th krb5kdc[2481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394240, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.194.195@KRBTEST.COM
I20260926 03:44:02.754726 3185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:02.740106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60116 (local address 127.0.194.195:39703)
0926 03:44:02.740612 (+ 506us) server_negotiation.cc:206] Beginning negotiation
0926 03:44:02.740617 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:44:02.740641 (+ 24us) server_negotiation.cc:407] Connection header received
0926 03:44:02.740692 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:44:02.740696 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:44:02.740756 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:44:02.740887 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:44:02.742143 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:02.742969 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:02.744316 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:02.744755 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:02.744897 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:02.749189 (+ 4292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:44:02.749204 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:44:02.749221 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:44:02.749255 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:44:02.752079 (+ 2824us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:02.752761 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:02.752769 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:02.752771 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:02.752897 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:02.753267 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:02.753274 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:02.753277 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:02.753610 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:44:02.753943 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0926 03:44:02.754161 (+ 218us) server_negotiation.cc:299] Negotiation successful
0926 03:44:02.754388 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":381,"thread_start_us":245,"threads_started":1}
W20260926 03:44:02.755492 3006 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60116
W20260926 03:44:02.756090 779 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu pid 2876: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 03:44:02.756242 779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu with pid 2876
I20260926 03:44:02.779599 779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu pid 2494
I20260926 03:44:02.780792 2566 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:44:02.896144 779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu with pid 2494
2026-09-26T03:44:02Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (8495 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790394221564982-779-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:44:02 dist-test-slave-n2th krb5kdc[3189](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:44:02 dist-test-slave-n2th krb5kdc[3189](info): set up 2 sockets
Sep 26 03:44:02 dist-test-slave-n2th krb5kdc[3189](info): commencing operation
krb5kdc: starting...
W20260926 03:44:03.979019 779 mini_kdc.cc:121] Time spent starting KDC: real 1.039s user 0.005s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790394221564982-779-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790394221564982-779-0/krb5kdc/test-user.keytab.
Sep 26 03:44:04 dist-test-slave-n2th krb5kdc[3189](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394244, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:44:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:44:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.194.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.194.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.194.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790394221564982-779-0/krb5kdc/kudu_127.0.194.254.keytab.
Entry for principal kudu/127.0.194.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790394221564982-779-0/krb5kdc/kudu_127.0.194.254.keytab.
I20260926 03:44:04.169646 779 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
/tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790394221564982-779-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790394221564982-779-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790394221564982-779-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790394221564982-779-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.194.254:37881
--webserver_interface=127.0.194.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790394221564982-779-0/krb5kdc/kudu_127.0.194.254.keytab
--principal=kudu/127.0.194.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.194.212:40327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.194.254:37881
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790394221564982-779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790394221564982-779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790394221564982-779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:44:04.485881 3202 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:44:04.486171 3202 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:44:04.486256 3202 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:44:04.493445 3202 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:44:04.493597 3202 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:44:04.493667 3202 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:44:04.493732 3202 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:44:04.493784 3202 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:44:04.503654 3202 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.194.212:40327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790394221564982-779-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790394221564982-779-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.194.254:37881
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790394221564982-779-0/krb5kdc/kudu_127.0.194.254.keytab
--principal=kudu/127.0.194.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.194.254:37881
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790394221564982-779-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.194.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790394221564982-779-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:44:04.504930 3202 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:44:04.506155 3202 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:44:04.519256 3207 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:44:04.519307 3210 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:44:04.519279 3208 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:44:04.520028 3202 server_base.cc:1034] running on GCE node
I20260926 03:44:04.520844 3202 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:44:04.522228 3202 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:44:04.523425 3202 hybrid_clock.cc:648] HybridClock initialized: now 1790394244523359 us; error 54 us; skew 500 ppm
I20260926 03:44:04.523841 3202 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:44:04 dist-test-slave-n2th krb5kdc[3189](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394244, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:44:04.526993 3202 init.cc:377] Logged in from keytab as kudu/127.0.194.254@KRBTEST.COM (short username kudu)
I20260926 03:44:04.528575 3202 webserver.cc:460] Webserver started at http://127.0.194.254:45539/ using document root <none> and password file <none>
I20260926 03:44:04.529311 3202 fs_manager.cc:359] Metadata directory not provided
I20260926 03:44:04.529451 3202 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:44:04.529779 3202 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:44:04.532815 3202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790394221564982-779-0/minicluster-data/master-0/data/instance:
uuid: "3804062ef17a47c08e94bfb1e827191b"
format_stamp: "Formatted at 2026-09-26 03:44:04 on dist-test-slave-n2th"
server_key: "e537afdb383003cdc0b4f869f6d05602"
server_key_iv: "4bec74406dcca6e0787120b137532c3c"
server_key_version: "encryptionkey@0"
I20260926 03:44:04.533605 3202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790394221564982-779-0/minicluster-data/master-0/wal/instance:
uuid: "3804062ef17a47c08e94bfb1e827191b"
format_stamp: "Formatted at 2026-09-26 03:44:04 on dist-test-slave-n2th"
server_key: "e537afdb383003cdc0b4f869f6d05602"
server_key_iv: "4bec74406dcca6e0787120b137532c3c"
server_key_version: "encryptionkey@0"
I20260926 03:44:04.538781 3202 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260926 03:44:04.542111 3217 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:44:04.543462 3202 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260926 03:44:04.543650 3202 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790394221564982-779-0/minicluster-data/master-0/data,/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790394221564982-779-0/minicluster-data/master-0/wal
uuid: "3804062ef17a47c08e94bfb1e827191b"
format_stamp: "Formatted at 2026-09-26 03:44:04 on dist-test-slave-n2th"
server_key: "e537afdb383003cdc0b4f869f6d05602"
server_key_iv: "4bec74406dcca6e0787120b137532c3c"
server_key_version: "encryptionkey@0"
I20260926 03:44:04.543843 3202 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790394221564982-779-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790394221564982-779-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790394221564982-779-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:44:04.560936 3202 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:44:04.564810 3202 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:44:04.565093 3202 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:44:04.583359 3202 rpc_server.cc:307] RPC server started. Bound to: 127.0.194.254:37881
I20260926 03:44:04.583382 3268 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.194.254:37881 every 8 connection(s)
I20260926 03:44:04.584791 3202 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790394221564982-779-0/minicluster-data/master-0/data/info.pb
I20260926 03:44:04.587141 779 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu as pid 3202
I20260926 03:44:04.587373 779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790394221564982-779-0/minicluster-data/master-0/wal/instance
I20260926 03:44:04.587968 779 external_mini_cluster.cc:1428] Setting key cf1d85f1121a29e7ea9ed243dcfa7c28
I20260926 03:44:04.588809 3269 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 26 03:44:04 dist-test-slave-n2th krb5kdc[3189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394244, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.194.254@KRBTEST.COM
I20260926 03:44:04.602567 3269 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3804062ef17a47c08e94bfb1e827191b: Bootstrap starting.
I20260926 03:44:04.606877 3269 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3804062ef17a47c08e94bfb1e827191b: Neither blocks nor log segments found. Creating new log.
I20260926 03:44:04.608183 3269 log.cc:824] T 00000000000000000000000000000000 P 3804062ef17a47c08e94bfb1e827191b: Log is configured to *not* fsync() on all Append() calls
I20260926 03:44:04.611563 3269 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3804062ef17a47c08e94bfb1e827191b: No bootstrap required, opened a new log
I20260926 03:44:04.611590 3272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:04.589977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40690 (local address 127.0.194.254:37881)
0926 03:44:04.590785 (+ 808us) server_negotiation.cc:206] Beginning negotiation
0926 03:44:04.590800 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 03:44:04.590849 (+ 49us) server_negotiation.cc:407] Connection header received
0926 03:44:04.592493 (+ 1644us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:44:04.592526 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:44:04.593040 (+ 514us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:44:04.593469 (+ 429us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:44:04.594779 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:04.596827 (+ 2048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:04.598088 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:04.599159 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:04.599359 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:04.603126 (+ 3767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:44:04.603136 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:44:04.603157 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:44:04.603200 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:44:04.607073 (+ 3873us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:04.607740 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:04.607748 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:04.607756 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:04.607978 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:04.608357 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:04.608364 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:04.608366 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:04.609086 (+ 720us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:44:04.609294 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0926 03:44:04.609839 (+ 545us) server_negotiation.cc:299] Negotiation successful
0926 03:44:04.610252 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":552,"thread_start_us":319,"threads_started":1}
I20260926 03:44:04.619678 3269 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3804062ef17a47c08e94bfb1e827191b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3804062ef17a47c08e94bfb1e827191b" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 37881 } }
I20260926 03:44:04.620028 3269 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3804062ef17a47c08e94bfb1e827191b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:44:04.620105 3269 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3804062ef17a47c08e94bfb1e827191b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3804062ef17a47c08e94bfb1e827191b, State: Initialized, Role: FOLLOWER
I20260926 03:44:04.620584 3269 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3804062ef17a47c08e94bfb1e827191b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3804062ef17a47c08e94bfb1e827191b" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 37881 } }
I20260926 03:44:04.620805 3269 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3804062ef17a47c08e94bfb1e827191b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:44:04.620918 3269 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3804062ef17a47c08e94bfb1e827191b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:44:04.621052 3269 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3804062ef17a47c08e94bfb1e827191b [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:44:04.623588 3269 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3804062ef17a47c08e94bfb1e827191b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3804062ef17a47c08e94bfb1e827191b" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 37881 } }
I20260926 03:44:04.624042 3269 leader_election.cc:302] T 00000000000000000000000000000000 P 3804062ef17a47c08e94bfb1e827191b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3804062ef17a47c08e94bfb1e827191b; no voters:
I20260926 03:44:04.624571 3269 leader_election.cc:288] T 00000000000000000000000000000000 P 3804062ef17a47c08e94bfb1e827191b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:44:04.624727 3274 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3804062ef17a47c08e94bfb1e827191b [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:44:04.625526 3274 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3804062ef17a47c08e94bfb1e827191b [term 1 LEADER]: Becoming Leader. State: Replica: 3804062ef17a47c08e94bfb1e827191b, State: Running, Role: LEADER
I20260926 03:44:04.626070 3274 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3804062ef17a47c08e94bfb1e827191b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3804062ef17a47c08e94bfb1e827191b" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 37881 } }
I20260926 03:44:04.626657 3269 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3804062ef17a47c08e94bfb1e827191b [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:44:04.629130 3276 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3804062ef17a47c08e94bfb1e827191b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3804062ef17a47c08e94bfb1e827191b. Latest consensus state: current_term: 1 leader_uuid: "3804062ef17a47c08e94bfb1e827191b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3804062ef17a47c08e94bfb1e827191b" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 37881 } } }
I20260926 03:44:04.629498 3276 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3804062ef17a47c08e94bfb1e827191b [sys.catalog]: This master's current role is: LEADER
I20260926 03:44:04.629494 3275 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3804062ef17a47c08e94bfb1e827191b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3804062ef17a47c08e94bfb1e827191b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3804062ef17a47c08e94bfb1e827191b" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 37881 } } }
I20260926 03:44:04.631443 3275 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3804062ef17a47c08e94bfb1e827191b [sys.catalog]: This master's current role is: LEADER
I20260926 03:44:04.632874 3280 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:44:04.637182 3280 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:44:04.645277 3280 catalog_manager.cc:1347] Generated new cluster ID: 601f25861dca4e1da84e2d5029469bd6
I20260926 03:44:04.645406 3280 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:44:04.680125 3280 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:44:04.681560 3280 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:44:04.688494 3280 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3804062ef17a47c08e94bfb1e827191b: Generated new TSK 0
I20260926 03:44:04.689306 3280 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.194.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.194.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.194.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790394221564982-779-0/krb5kdc/kudu_127.0.194.193.keytab.
Entry for principal kudu/127.0.194.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790394221564982-779-0/krb5kdc/kudu_127.0.194.193.keytab.
I20260926 03:44:04.743213 779 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
/tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790394221564982-779-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790394221564982-779-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790394221564982-779-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790394221564982-779-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.194.193:0
--local_ip_for_outbound_sockets=127.0.194.193
--webserver_interface=127.0.194.193
--webserver_port=0
--tserver_master_addrs=127.0.194.254:37881
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790394221564982-779-0/krb5kdc/kudu_127.0.194.193.keytab
--principal=kudu/127.0.194.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.194.212:40327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790394221564982-779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790394221564982-779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790394221564982-779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:44:05.092809 3295 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:44:05.093115 3295 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:44:05.093294 3295 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:44:05.100648 3295 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:44:05.100771 3295 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:44:05.101003 3295 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.194.193
I20260926 03:44:05.111248 3295 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.194.212:40327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790394221564982-779-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790394221564982-779-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790394221564982-779-0/krb5kdc/kudu_127.0.194.193.keytab
--principal=kudu/127.0.194.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.194.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790394221564982-779-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.194.193
--webserver_port=0
--tserver_master_addrs=127.0.194.254:37881
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.194.193
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790394221564982-779-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:44:05.112707 3295 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:44:05.114153 3295 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:44:05.127797 3300 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:44:05.127712 3303 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:44:05.127779 3301 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:44:05.128648 3295 server_base.cc:1034] running on GCE node
I20260926 03:44:05.129387 3295 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:44:05.130419 3295 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:44:05.131728 3295 hybrid_clock.cc:648] HybridClock initialized: now 1790394245131644 us; error 74 us; skew 500 ppm
I20260926 03:44:05.132136 3295 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:44:05 dist-test-slave-n2th krb5kdc[3189](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394245, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:44:05.135535 3295 init.cc:377] Logged in from keytab as kudu/127.0.194.193@KRBTEST.COM (short username kudu)
I20260926 03:44:05.137106 3295 webserver.cc:460] Webserver started at http://127.0.194.193:43871/ using document root <none> and password file <none>
I20260926 03:44:05.137804 3295 fs_manager.cc:359] Metadata directory not provided
I20260926 03:44:05.137931 3295 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:44:05.138257 3295 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:44:05.141239 3295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790394221564982-779-0/minicluster-data/ts-0/data/instance:
uuid: "3b10cb05980d49989848f1acdbe905cd"
format_stamp: "Formatted at 2026-09-26 03:44:05 on dist-test-slave-n2th"
server_key: "2402389e2ee2842a623dcc62e4eb9656"
server_key_iv: "b52ca395df5cec7b5d5e30c29ab8acda"
server_key_version: "encryptionkey@0"
I20260926 03:44:05.142055 3295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790394221564982-779-0/minicluster-data/ts-0/wal/instance:
uuid: "3b10cb05980d49989848f1acdbe905cd"
format_stamp: "Formatted at 2026-09-26 03:44:05 on dist-test-slave-n2th"
server_key: "2402389e2ee2842a623dcc62e4eb9656"
server_key_iv: "b52ca395df5cec7b5d5e30c29ab8acda"
server_key_version: "encryptionkey@0"
I20260926 03:44:05.147411 3295 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260926 03:44:05.150982 3310 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:44:05.152467 3295 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 03:44:05.152665 3295 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790394221564982-779-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790394221564982-779-0/minicluster-data/ts-0/wal
uuid: "3b10cb05980d49989848f1acdbe905cd"
format_stamp: "Formatted at 2026-09-26 03:44:05 on dist-test-slave-n2th"
server_key: "2402389e2ee2842a623dcc62e4eb9656"
server_key_iv: "b52ca395df5cec7b5d5e30c29ab8acda"
server_key_version: "encryptionkey@0"
I20260926 03:44:05.152892 3295 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790394221564982-779-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790394221564982-779-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790394221564982-779-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:44:05.170403 3295 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:44:05.174551 3295 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:44:05.174887 3295 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:44:05.175899 3295 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:44:05.177251 3295 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:44:05.177335 3295 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:44:05.177518 3295 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:44:05.177579 3295 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:44:05.211294 3295 rpc_server.cc:307] RPC server started. Bound to: 127.0.194.193:44809
I20260926 03:44:05.211351 3422 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.194.193:44809 every 8 connection(s)
I20260926 03:44:05.212579 3295 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790394221564982-779-0/minicluster-data/ts-0/data/info.pb
I20260926 03:44:05.222817 779 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu as pid 3295
I20260926 03:44:05.223030 779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790394221564982-779-0/minicluster-data/ts-0/wal/instance
I20260926 03:44:05.223793 779 external_mini_cluster.cc:1428] Setting key 0e2812b404c8ae004817e648cec1bc7c
Sep 26 03:44:05 dist-test-slave-n2th krb5kdc[3189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394245, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.193@KRBTEST.COM for kudu/127.0.194.254@KRBTEST.COM
I20260926 03:44:05.232630 3425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:05.215115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.193:57237 (local address 127.0.194.254:37881)
0926 03:44:05.215551 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0926 03:44:05.215555 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:44:05.216432 (+ 877us) server_negotiation.cc:407] Connection header received
0926 03:44:05.218222 (+ 1790us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:44:05.218228 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:44:05.218316 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:44:05.218440 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:44:05.219946 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:05.220641 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:05.222187 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:05.222580 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:05.222719 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:05.226596 (+ 3877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:44:05.226610 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:44:05.226614 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:44:05.226647 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:44:05.229415 (+ 2768us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:05.230173 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:05.230179 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:05.230187 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:05.230300 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:05.230782 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:05.230787 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:05.230791 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:05.231063 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:44:05.231218 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0926 03:44:05.232003 (+ 785us) server_negotiation.cc:299] Negotiation successful
0926 03:44:05.232237 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":234,"threads_started":1}
I20260926 03:44:05.233660 3426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:05.215580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:37881 (local address 127.0.194.193:57237)
0926 03:44:05.216263 (+ 683us) negotiation.cc:107] Waiting for socket to connect
0926 03:44:05.216307 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 03:44:05.217959 (+ 1652us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:44:05.218669 (+ 710us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:44:05.218678 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:44:05.219241 (+ 563us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:44:05.219772 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:05.219789 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:05.220801 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:05.220806 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:05.222035 (+ 1229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:05.222046 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:05.222832 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:05.222840 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:05.223156 (+ 316us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:05.223954 (+ 798us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:44:05.223989 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:44:05.226360 (+ 2371us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:44:05.229575 (+ 3215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:44:05.229585 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:44:05.229603 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:44:05.230036 (+ 433us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:44:05.230415 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:44:05.230421 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:44:05.230424 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:44:05.230642 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:44:05.231207 (+ 565us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:44:05.231215 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:44:05.231579 (+ 364us) client_negotiation.cc:769] Sending connection context
0926 03:44:05.231906 (+ 327us) client_negotiation.cc:240] Negotiation successful
0926 03:44:05.232503 (+ 597us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":268,"threads_started":1}
I20260926 03:44:05.235304 3423 heartbeater.cc:347] Connected to a master server at 127.0.194.254:37881
I20260926 03:44:05.235634 3423 heartbeater.cc:464] Registering TS with master...
I20260926 03:44:05.236516 3423 heartbeater.cc:511] Master 127.0.194.254:37881 requested a full tablet report, sending...
I20260926 03:44:05.239037 3234 ts_manager.cc:194] Registered new tserver with Master: 3b10cb05980d49989848f1acdbe905cd (127.0.194.193:44809)
I20260926 03:44:05.241294 3234 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.194.193@KRBTEST.COM'} at 127.0.194.193:57237
WARNING: no policy specified for kudu/127.0.194.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.194.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.194.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790394221564982-779-0/krb5kdc/kudu_127.0.194.194.keytab.
Entry for principal kudu/127.0.194.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790394221564982-779-0/krb5kdc/kudu_127.0.194.194.keytab.
I20260926 03:44:05.270164 779 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
/tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790394221564982-779-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790394221564982-779-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790394221564982-779-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790394221564982-779-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.194.194:0
--local_ip_for_outbound_sockets=127.0.194.194
--webserver_interface=127.0.194.194
--webserver_port=0
--tserver_master_addrs=127.0.194.254:37881
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790394221564982-779-0/krb5kdc/kudu_127.0.194.194.keytab
--principal=kudu/127.0.194.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.194.212:40327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790394221564982-779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790394221564982-779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790394221564982-779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:44:05.589375 3429 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:44:05.589651 3429 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:44:05.589818 3429 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:44:05.597184 3429 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:44:05.597298 3429 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:44:05.597491 3429 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.194.194
I20260926 03:44:05.608086 3429 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.194.212:40327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790394221564982-779-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790394221564982-779-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790394221564982-779-0/krb5kdc/kudu_127.0.194.194.keytab
--principal=kudu/127.0.194.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.194.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790394221564982-779-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.194.194
--webserver_port=0
--tserver_master_addrs=127.0.194.254:37881
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.194.194
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790394221564982-779-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:44:05.609460 3429 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:44:05.610823 3429 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:44:05.623919 3434 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:44:05.624010 3435 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:44:05.624051 3437 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:44:05.624893 3429 server_base.cc:1034] running on GCE node
I20260926 03:44:05.625640 3429 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:44:05.626672 3429 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:44:05.627940 3429 hybrid_clock.cc:648] HybridClock initialized: now 1790394245627885 us; error 43 us; skew 500 ppm
I20260926 03:44:05.628305 3429 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:44:05 dist-test-slave-n2th krb5kdc[3189](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394245, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:44:05.631713 3429 init.cc:377] Logged in from keytab as kudu/127.0.194.194@KRBTEST.COM (short username kudu)
I20260926 03:44:05.633653 3429 webserver.cc:460] Webserver started at http://127.0.194.194:36925/ using document root <none> and password file <none>
I20260926 03:44:05.634377 3429 fs_manager.cc:359] Metadata directory not provided
I20260926 03:44:05.634527 3429 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:44:05.634918 3429 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:44:05.637930 3429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790394221564982-779-0/minicluster-data/ts-1/data/instance:
uuid: "4280c4449224483193111a8ea398ebe1"
format_stamp: "Formatted at 2026-09-26 03:44:05 on dist-test-slave-n2th"
server_key: "936b8f1777224105af6fe05dad8f4059"
server_key_iv: "bb8093d92508a38d2fe2f15d9da248a3"
server_key_version: "encryptionkey@0"
I20260926 03:44:05.638948 3429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790394221564982-779-0/minicluster-data/ts-1/wal/instance:
uuid: "4280c4449224483193111a8ea398ebe1"
format_stamp: "Formatted at 2026-09-26 03:44:05 on dist-test-slave-n2th"
server_key: "936b8f1777224105af6fe05dad8f4059"
server_key_iv: "bb8093d92508a38d2fe2f15d9da248a3"
server_key_version: "encryptionkey@0"
I20260926 03:44:05.644809 3429 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 03:44:05.648520 3444 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:44:05.649894 3429 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260926 03:44:05.650074 3429 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790394221564982-779-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790394221564982-779-0/minicluster-data/ts-1/wal
uuid: "4280c4449224483193111a8ea398ebe1"
format_stamp: "Formatted at 2026-09-26 03:44:05 on dist-test-slave-n2th"
server_key: "936b8f1777224105af6fe05dad8f4059"
server_key_iv: "bb8093d92508a38d2fe2f15d9da248a3"
server_key_version: "encryptionkey@0"
I20260926 03:44:05.650282 3429 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790394221564982-779-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790394221564982-779-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790394221564982-779-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:44:05.674656 3429 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:44:05.679283 3429 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:44:05.679607 3429 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:44:05.680603 3429 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:44:05.681929 3429 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:44:05.682034 3429 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:44:05.682185 3429 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:44:05.682257 3429 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:44:05.716267 3429 rpc_server.cc:307] RPC server started. Bound to: 127.0.194.194:40593
I20260926 03:44:05.716475 3556 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.194.194:40593 every 8 connection(s)
I20260926 03:44:05.717912 3429 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790394221564982-779-0/minicluster-data/ts-1/data/info.pb
I20260926 03:44:05.719188 779 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu as pid 3429
I20260926 03:44:05.719537 779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790394221564982-779-0/minicluster-data/ts-1/wal/instance
I20260926 03:44:05.721256 779 external_mini_cluster.cc:1428] Setting key b941a53d5d086b2f8545ca7787a56a73
Sep 26 03:44:05 dist-test-slave-n2th krb5kdc[3189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394245, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.194@KRBTEST.COM for kudu/127.0.194.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.194.195@KRBTEST.COM; defaulting to no policy
I20260926 03:44:05.742439 3425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:05.721442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.194:57611 (local address 127.0.194.254:37881)
0926 03:44:05.722426 (+ 984us) server_negotiation.cc:206] Beginning negotiation
0926 03:44:05.722433 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:44:05.723562 (+ 1129us) server_negotiation.cc:407] Connection header received
0926 03:44:05.725687 (+ 2125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:44:05.725692 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:44:05.725780 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:44:05.725874 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:44:05.727861 (+ 1987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:05.729056 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:05.731466 (+ 2410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:05.731957 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:05.732063 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:05.736418 (+ 4355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:44:05.736423 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:44:05.736427 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:44:05.736461 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:44:05.739274 (+ 2813us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:05.740057 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:05.740063 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:05.740066 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:05.740195 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:05.740637 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:05.740642 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:05.740649 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:05.740913 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:44:05.741070 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0926 03:44:05.741855 (+ 785us) server_negotiation.cc:299] Negotiation successful
0926 03:44:05.742086 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":843}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.194.195@KRBTEST.COM" created.
I20260926 03:44:05.743888 3559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:05.721909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:37881 (local address 127.0.194.194:57611)
0926 03:44:05.723394 (+ 1485us) negotiation.cc:107] Waiting for socket to connect
0926 03:44:05.723445 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 03:44:05.725387 (+ 1942us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:44:05.726155 (+ 768us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:44:05.726163 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:44:05.726841 (+ 678us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:44:05.727670 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:05.727695 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:05.729215 (+ 1520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:05.729222 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:05.731303 (+ 2081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:05.731323 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:05.732198 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:05.732208 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:05.732549 (+ 341us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:05.733339 (+ 790us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:44:05.733375 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:44:05.736147 (+ 2772us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:44:05.739434 (+ 3287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:44:05.739443 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:44:05.739463 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:44:05.739913 (+ 450us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:44:05.740310 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:44:05.740315 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:44:05.740319 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:44:05.740516 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:44:05.741055 (+ 539us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:44:05.741064 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:44:05.741479 (+ 415us) client_negotiation.cc:769] Sending connection context
0926 03:44:05.741754 (+ 275us) client_negotiation.cc:240] Negotiation successful
0926 03:44:05.742328 (+ 574us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1169,"thread_start_us":935,"threads_started":1}
I20260926 03:44:05.746160 3557 heartbeater.cc:347] Connected to a master server at 127.0.194.254:37881
I20260926 03:44:05.746510 3557 heartbeater.cc:464] Registering TS with master...
I20260926 03:44:05.747473 3557 heartbeater.cc:511] Master 127.0.194.254:37881 requested a full tablet report, sending...
I20260926 03:44:05.749089 3234 ts_manager.cc:194] Registered new tserver with Master: 4280c4449224483193111a8ea398ebe1 (127.0.194.194:40593)
I20260926 03:44:05.750303 3234 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.194.194@KRBTEST.COM'} at 127.0.194.194:57611
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.194.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790394221564982-779-0/krb5kdc/kudu_127.0.194.195.keytab.
Entry for principal kudu/127.0.194.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790394221564982-779-0/krb5kdc/kudu_127.0.194.195.keytab.
I20260926 03:44:05.769380 779 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
/tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790394221564982-779-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790394221564982-779-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790394221564982-779-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790394221564982-779-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.194.195:0
--local_ip_for_outbound_sockets=127.0.194.195
--webserver_interface=127.0.194.195
--webserver_port=0
--tserver_master_addrs=127.0.194.254:37881
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790394221564982-779-0/krb5kdc/kudu_127.0.194.195.keytab
--principal=kudu/127.0.194.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.194.212:40327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790394221564982-779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790394221564982-779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790394221564982-779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:44:06.092425 3562 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:44:06.092727 3562 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:44:06.092911 3562 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:44:06.100354 3562 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:44:06.100476 3562 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:44:06.100668 3562 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.194.195
I20260926 03:44:06.110675 3562 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.194.212:40327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790394221564982-779-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790394221564982-779-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790394221564982-779-0/krb5kdc/kudu_127.0.194.195.keytab
--principal=kudu/127.0.194.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.194.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790394221564982-779-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.194.195
--webserver_port=0
--tserver_master_addrs=127.0.194.254:37881
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.194.195
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790394221564982-779-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:44:06.112077 3562 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:44:06.113425 3562 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:44:06.126857 3567 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:44:06.126957 3568 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:44:06.126981 3570 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:44:06.127941 3562 server_base.cc:1034] running on GCE node
I20260926 03:44:06.128621 3562 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:44:06.129695 3562 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:44:06.130918 3562 hybrid_clock.cc:648] HybridClock initialized: now 1790394246130851 us; error 41 us; skew 500 ppm
I20260926 03:44:06.131261 3562 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:44:06 dist-test-slave-n2th krb5kdc[3189](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394246, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:44:06.134586 3562 init.cc:377] Logged in from keytab as kudu/127.0.194.195@KRBTEST.COM (short username kudu)
I20260926 03:44:06.136238 3562 webserver.cc:460] Webserver started at http://127.0.194.195:43977/ using document root <none> and password file <none>
I20260926 03:44:06.136976 3562 fs_manager.cc:359] Metadata directory not provided
I20260926 03:44:06.137130 3562 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:44:06.137451 3562 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:44:06.140637 3562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790394221564982-779-0/minicluster-data/ts-2/data/instance:
uuid: "fea2df6d512c4b4bbced42d42b414214"
format_stamp: "Formatted at 2026-09-26 03:44:06 on dist-test-slave-n2th"
server_key: "6c87ee08fbfc901d5abe8418c636c680"
server_key_iv: "7031d5998df881ed2e9c4cccb0f24641"
server_key_version: "encryptionkey@0"
I20260926 03:44:06.141453 3562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790394221564982-779-0/minicluster-data/ts-2/wal/instance:
uuid: "fea2df6d512c4b4bbced42d42b414214"
format_stamp: "Formatted at 2026-09-26 03:44:06 on dist-test-slave-n2th"
server_key: "6c87ee08fbfc901d5abe8418c636c680"
server_key_iv: "7031d5998df881ed2e9c4cccb0f24641"
server_key_version: "encryptionkey@0"
I20260926 03:44:06.146836 3562 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260926 03:44:06.150677 3577 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:44:06.152117 3562 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260926 03:44:06.152309 3562 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790394221564982-779-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790394221564982-779-0/minicluster-data/ts-2/wal
uuid: "fea2df6d512c4b4bbced42d42b414214"
format_stamp: "Formatted at 2026-09-26 03:44:06 on dist-test-slave-n2th"
server_key: "6c87ee08fbfc901d5abe8418c636c680"
server_key_iv: "7031d5998df881ed2e9c4cccb0f24641"
server_key_version: "encryptionkey@0"
I20260926 03:44:06.152531 3562 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790394221564982-779-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790394221564982-779-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790394221564982-779-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:44:06.167543 3562 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:44:06.171777 3562 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:44:06.172097 3562 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:44:06.173100 3562 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:44:06.174482 3562 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:44:06.174599 3562 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:44:06.174788 3562 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:44:06.174882 3562 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260926 03:44:06.208807 3562 rpc_server.cc:307] RPC server started. Bound to: 127.0.194.195:37793
I20260926 03:44:06.208849 3689 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.194.195:37793 every 8 connection(s)
I20260926 03:44:06.210175 3562 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790394221564982-779-0/minicluster-data/ts-2/data/info.pb
I20260926 03:44:06.218040 779 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu as pid 3562
I20260926 03:44:06.218245 779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790394221564982-779-0/minicluster-data/ts-2/wal/instance
I20260926 03:44:06.218833 779 external_mini_cluster.cc:1428] Setting key 46adc422d1d6ba377094ae32ec1cecaa
Sep 26 03:44:06 dist-test-slave-n2th krb5kdc[3189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394246, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.195@KRBTEST.COM for kudu/127.0.194.254@KRBTEST.COM
I20260926 03:44:06.231792 3425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:06.213090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.195:37479 (local address 127.0.194.254:37881)
0926 03:44:06.213273 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0926 03:44:06.213278 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:44:06.214710 (+ 1432us) server_negotiation.cc:407] Connection header received
0926 03:44:06.216578 (+ 1868us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:44:06.216584 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:44:06.216700 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:44:06.216820 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:44:06.218421 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:06.219516 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:06.221355 (+ 1839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:06.221908 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:06.222045 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:06.225795 (+ 3750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:44:06.225802 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:44:06.225806 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:44:06.225849 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:44:06.228577 (+ 2728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:06.229357 (+ 780us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:06.229363 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:06.229367 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:06.229507 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:06.229956 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:06.229963 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:06.229966 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:06.230249 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:44:06.230407 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0926 03:44:06.231218 (+ 811us) server_negotiation.cc:299] Negotiation successful
0926 03:44:06.231435 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 03:44:06.233017 3692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:06.213691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:37881 (local address 127.0.194.195:37479)
0926 03:44:06.214505 (+ 814us) negotiation.cc:107] Waiting for socket to connect
0926 03:44:06.214564 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0926 03:44:06.216282 (+ 1718us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:44:06.217061 (+ 779us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:44:06.217069 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:44:06.217571 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:44:06.218206 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:06.218231 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:06.219705 (+ 1474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:06.219711 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:06.221193 (+ 1482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:06.221205 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:06.222146 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:06.222155 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:06.222412 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:06.223068 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:44:06.223091 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:44:06.225524 (+ 2433us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:44:06.228737 (+ 3213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:44:06.228747 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:44:06.228767 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:44:06.229215 (+ 448us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:44:06.229630 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:44:06.229636 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:44:06.229639 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:44:06.229839 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:44:06.230384 (+ 545us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:44:06.230393 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:44:06.230839 (+ 446us) client_negotiation.cc:769] Sending connection context
0926 03:44:06.231126 (+ 287us) client_negotiation.cc:240] Negotiation successful
0926 03:44:06.231696 (+ 570us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":537,"thread_start_us":327,"threads_started":1}
I20260926 03:44:06.234543 3690 heartbeater.cc:347] Connected to a master server at 127.0.194.254:37881
I20260926 03:44:06.234894 3690 heartbeater.cc:464] Registering TS with master...
I20260926 03:44:06.235579 3690 heartbeater.cc:511] Master 127.0.194.254:37881 requested a full tablet report, sending...
I20260926 03:44:06.236976 3233 ts_manager.cc:194] Registered new tserver with Master: fea2df6d512c4b4bbced42d42b414214 (127.0.194.195:37793)
I20260926 03:44:06.237864 3233 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.194.195@KRBTEST.COM'} at 127.0.194.195:37479
I20260926 03:44:06.244863 3423 heartbeater.cc:503] Master 127.0.194.254:37881 was elected leader, sending a full tablet report...
I20260926 03:44:06.247164 779 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260926 03:44:06.278133 3425 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:44:06.275496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40696 (local address 127.0.194.254:37881)
0926 03:44:06.275669 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0926 03:44:06.275693 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0926 03:44:06.276181 (+ 488us) server_negotiation.cc:407] Connection header received
0926 03:44:06.276409 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:44:06.276413 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:44:06.276474 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:44:06.276581 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:44:06.277844 (+ 1263us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40696: BlockingRecv error: recv got EOF from 127.0.0.1:40696 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 03:44:06.281251 779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu pid 3295
W20260926 03:44:06.284236 779 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu pid 3295: Not authorized: Client connection negotiation failed: client connection to 127.0.194.193:44809: server requires authentication, but client does not have Kerberos credentials available
I20260926 03:44:06.284408 779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu with pid 3295
I20260926 03:44:06.299918 779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu pid 3429
W20260926 03:44:06.302505 779 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu pid 3429: Not authorized: Client connection negotiation failed: client connection to 127.0.194.194:40593: server requires authentication, but client does not have Kerberos credentials available
I20260926 03:44:06.302678 779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu with pid 3429
I20260926 03:44:06.317925 779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu pid 3562
W20260926 03:44:06.320917 779 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu pid 3562: Not authorized: Client connection negotiation failed: client connection to 127.0.194.195:37793: server requires authentication, but client does not have Kerberos credentials available
I20260926 03:44:06.321070 779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu with pid 3562
I20260926 03:44:06.336140 779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu pid 3202
I20260926 03:44:06.337272 3264 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:44:06.438135 779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu with pid 3202
2026-09-26T03:44:06Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3539 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790394221564982-779-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:44:06 dist-test-slave-n2th krb5kdc[3707](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:44:06 dist-test-slave-n2th krb5kdc[3707](info): set up 2 sockets
Sep 26 03:44:06 dist-test-slave-n2th krb5kdc[3707](info): commencing operation
krb5kdc: starting...
W20260926 03:44:07.511166 779 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.000s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790394221564982-779-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790394221564982-779-0/krb5kdc/test-user.keytab.
Sep 26 03:44:07 dist-test-slave-n2th krb5kdc[3707](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394247, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:44:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:44:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.194.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.194.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.194.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790394221564982-779-0/krb5kdc/kudu_127.0.194.254.keytab.
Entry for principal kudu/127.0.194.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790394221564982-779-0/krb5kdc/kudu_127.0.194.254.keytab.
I20260926 03:44:07.708563 779 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
/tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790394221564982-779-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790394221564982-779-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790394221564982-779-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790394221564982-779-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.194.254:45975
--webserver_interface=127.0.194.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790394221564982-779-0/krb5kdc/kudu_127.0.194.254.keytab
--principal=kudu/127.0.194.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.194.212:33139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.194.254:45975
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790394221564982-779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790394221564982-779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790394221564982-779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:44:08.037505 3720 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:44:08.037812 3720 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:44:08.038003 3720 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:44:08.045544 3720 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:44:08.045670 3720 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:44:08.045784 3720 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:44:08.045895 3720 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:44:08.045979 3720 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:44:08.055883 3720 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.194.212:33139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790394221564982-779-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790394221564982-779-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.194.254:45975
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790394221564982-779-0/krb5kdc/kudu_127.0.194.254.keytab
--principal=kudu/127.0.194.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.194.254:45975
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790394221564982-779-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.194.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790394221564982-779-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:44:08.057230 3720 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:44:08.058533 3720 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:44:08.070439 3725 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:44:08.070420 3726 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:44:08.070945 3728 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:44:08.071241 3720 server_base.cc:1034] running on GCE node
I20260926 03:44:08.072059 3720 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:44:08.073593 3720 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:44:08.074837 3720 hybrid_clock.cc:648] HybridClock initialized: now 1790394248074777 us; error 49 us; skew 500 ppm
I20260926 03:44:08.075276 3720 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:44:08 dist-test-slave-n2th krb5kdc[3707](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394248, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:44:08.084306 3720 init.cc:377] Logged in from keytab as kudu/127.0.194.254@KRBTEST.COM (short username kudu)
I20260926 03:44:08.085918 3720 webserver.cc:460] Webserver started at http://127.0.194.254:40097/ using document root <none> and password file <none>
I20260926 03:44:08.086647 3720 fs_manager.cc:359] Metadata directory not provided
I20260926 03:44:08.086818 3720 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:44:08.087139 3720 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:44:08.090090 3720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790394221564982-779-0/minicluster-data/master-0/data/instance:
uuid: "34b8b33462c94df18ccad72f9c583456"
format_stamp: "Formatted at 2026-09-26 03:44:08 on dist-test-slave-n2th"
server_key: "757d624644b4bafb1ea511c397f1bd93"
server_key_iv: "02cf8499db6d97178c7b08b6e8dd81ea"
server_key_version: "encryptionkey@0"
I20260926 03:44:08.090979 3720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790394221564982-779-0/minicluster-data/master-0/wal/instance:
uuid: "34b8b33462c94df18ccad72f9c583456"
format_stamp: "Formatted at 2026-09-26 03:44:08 on dist-test-slave-n2th"
server_key: "757d624644b4bafb1ea511c397f1bd93"
server_key_iv: "02cf8499db6d97178c7b08b6e8dd81ea"
server_key_version: "encryptionkey@0"
I20260926 03:44:08.096206 3720 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 03:44:08.099673 3735 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:44:08.100961 3720 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 03:44:08.101148 3720 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790394221564982-779-0/minicluster-data/master-0/data,/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790394221564982-779-0/minicluster-data/master-0/wal
uuid: "34b8b33462c94df18ccad72f9c583456"
format_stamp: "Formatted at 2026-09-26 03:44:08 on dist-test-slave-n2th"
server_key: "757d624644b4bafb1ea511c397f1bd93"
server_key_iv: "02cf8499db6d97178c7b08b6e8dd81ea"
server_key_version: "encryptionkey@0"
I20260926 03:44:08.101346 3720 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790394221564982-779-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790394221564982-779-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790394221564982-779-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:44:08.114898 3720 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:44:08.119499 3720 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:44:08.119809 3720 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:44:08.138837 3720 rpc_server.cc:307] RPC server started. Bound to: 127.0.194.254:45975
I20260926 03:44:08.138887 3786 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.194.254:45975 every 8 connection(s)
I20260926 03:44:08.140359 3720 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790394221564982-779-0/minicluster-data/master-0/data/info.pb
I20260926 03:44:08.144364 3787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:44:08.150480 779 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu as pid 3720
I20260926 03:44:08.150779 779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790394221564982-779-0/minicluster-data/master-0/wal/instance
I20260926 03:44:08.151465 779 external_mini_cluster.cc:1428] Setting key 5f57486c6e9e90d1348f3be9bddb97b9
I20260926 03:44:08.156152 3787 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34b8b33462c94df18ccad72f9c583456: Bootstrap starting.
I20260926 03:44:08.160175 3787 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 34b8b33462c94df18ccad72f9c583456: Neither blocks nor log segments found. Creating new log.
I20260926 03:44:08.161501 3787 log.cc:824] T 00000000000000000000000000000000 P 34b8b33462c94df18ccad72f9c583456: Log is configured to *not* fsync() on all Append() calls
I20260926 03:44:08.164870 3787 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34b8b33462c94df18ccad72f9c583456: No bootstrap required, opened a new log
Sep 26 03:44:08 dist-test-slave-n2th krb5kdc[3707](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394247, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.194.254@KRBTEST.COM
I20260926 03:44:08.175194 3787 raft_consensus.cc:348] T 00000000000000000000000000000000 P 34b8b33462c94df18ccad72f9c583456 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34b8b33462c94df18ccad72f9c583456" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 45975 } }
I20260926 03:44:08.175726 3787 raft_consensus.cc:374] T 00000000000000000000000000000000 P 34b8b33462c94df18ccad72f9c583456 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:44:08.175846 3787 raft_consensus.cc:729] T 00000000000000000000000000000000 P 34b8b33462c94df18ccad72f9c583456 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34b8b33462c94df18ccad72f9c583456, State: Initialized, Role: FOLLOWER
I20260926 03:44:08.176502 3787 consensus_queue.cc:260] T 00000000000000000000000000000000 P 34b8b33462c94df18ccad72f9c583456 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34b8b33462c94df18ccad72f9c583456" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 45975 } }
I20260926 03:44:08.176780 3787 raft_consensus.cc:388] T 00000000000000000000000000000000 P 34b8b33462c94df18ccad72f9c583456 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:44:08.176931 3787 raft_consensus.cc:482] T 00000000000000000000000000000000 P 34b8b33462c94df18ccad72f9c583456 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:44:08.177125 3787 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 34b8b33462c94df18ccad72f9c583456 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:44:08.177459 3790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:08.153463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38296 (local address 127.0.194.254:45975)
0926 03:44:08.154295 (+ 832us) server_negotiation.cc:206] Beginning negotiation
0926 03:44:08.154310 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 03:44:08.154360 (+ 50us) server_negotiation.cc:407] Connection header received
0926 03:44:08.156795 (+ 2435us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:44:08.156825 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:44:08.157343 (+ 518us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:44:08.157816 (+ 473us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:44:08.159305 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:08.161293 (+ 1988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:08.163013 (+ 1720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:08.164037 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:08.164231 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:08.168351 (+ 4120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:44:08.168362 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:44:08.168388 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:44:08.168442 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:44:08.172626 (+ 4184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:08.173395 (+ 769us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:08.173407 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:08.173420 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:08.173639 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:08.174020 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:08.174025 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:08.174028 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:08.174903 (+ 875us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:44:08.175216 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0926 03:44:08.175726 (+ 510us) server_negotiation.cc:299] Negotiation successful
0926 03:44:08.176202 (+ 476us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":582,"thread_start_us":361,"threads_started":1}
I20260926 03:44:08.179782 3787 raft_consensus.cc:504] T 00000000000000000000000000000000 P 34b8b33462c94df18ccad72f9c583456 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34b8b33462c94df18ccad72f9c583456" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 45975 } }
I20260926 03:44:08.180337 3787 leader_election.cc:302] T 00000000000000000000000000000000 P 34b8b33462c94df18ccad72f9c583456 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 34b8b33462c94df18ccad72f9c583456; no voters:
I20260926 03:44:08.181046 3787 leader_election.cc:288] T 00000000000000000000000000000000 P 34b8b33462c94df18ccad72f9c583456 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:44:08.181198 3792 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 34b8b33462c94df18ccad72f9c583456 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:44:08.181401 779 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 03:44:08.182381 3792 raft_consensus.cc:686] T 00000000000000000000000000000000 P 34b8b33462c94df18ccad72f9c583456 [term 1 LEADER]: Becoming Leader. State: Replica: 34b8b33462c94df18ccad72f9c583456, State: Running, Role: LEADER
I20260926 03:44:08.182943 3792 consensus_queue.cc:237] T 00000000000000000000000000000000 P 34b8b33462c94df18ccad72f9c583456 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34b8b33462c94df18ccad72f9c583456" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 45975 } }
I20260926 03:44:08.183482 3787 sys_catalog.cc:564] T 00000000000000000000000000000000 P 34b8b33462c94df18ccad72f9c583456 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:44:08.185948 3793 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34b8b33462c94df18ccad72f9c583456 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "34b8b33462c94df18ccad72f9c583456" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34b8b33462c94df18ccad72f9c583456" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 45975 } } }
I20260926 03:44:08.186354 3793 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34b8b33462c94df18ccad72f9c583456 [sys.catalog]: This master's current role is: LEADER
I20260926 03:44:08.187374 3794 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34b8b33462c94df18ccad72f9c583456 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 34b8b33462c94df18ccad72f9c583456. Latest consensus state: current_term: 1 leader_uuid: "34b8b33462c94df18ccad72f9c583456" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34b8b33462c94df18ccad72f9c583456" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 45975 } } }
I20260926 03:44:08.187767 3794 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34b8b33462c94df18ccad72f9c583456 [sys.catalog]: This master's current role is: LEADER
I20260926 03:44:08.189975 3797 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:44:08.195086 3797 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:44:08.213603 3797 catalog_manager.cc:1347] Generated new cluster ID: 4d1f7fe7516e4a28ac5eabaaed21663a
I20260926 03:44:08.213783 3797 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:44:08.231721 3813 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260926 03:44:08.244453 3790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:08.235040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38312 (local address 127.0.194.254:45975)
0926 03:44:08.235241 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0926 03:44:08.235248 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:44:08.235857 (+ 609us) server_negotiation.cc:407] Connection header received
0926 03:44:08.236060 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:44:08.236066 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:44:08.236158 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:44:08.236269 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:44:08.237561 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:08.238673 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:08.240608 (+ 1935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:08.241276 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:08.241395 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:08.242696 (+ 1301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:44:08.242703 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:44:08.242707 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:44:08.243621 (+ 914us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:44:08.243689 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:44:08.243716 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:44:08.243822 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0926 03:44:08.243994 (+ 172us) server_negotiation.cc:299] Negotiation successful
0926 03:44:08.244090 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260926 03:44:08.256840 3797 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:44:08.258229 3797 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:44:08.268601 3797 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 34b8b33462c94df18ccad72f9c583456: Generated new TSK 0
I20260926 03:44:08.269730 3797 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 03:44:08.283164 779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu pid 3720
I20260926 03:44:08.284574 3782 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:44:08.376714 779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu with pid 3720
2026-09-26T03:44:08Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1932 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790394221564982-779-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:44:08 dist-test-slave-n2th krb5kdc[3821](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:44:08 dist-test-slave-n2th krb5kdc[3821](info): set up 2 sockets
Sep 26 03:44:08 dist-test-slave-n2th krb5kdc[3821](info): commencing operation
krb5kdc: starting...
W20260926 03:44:09.475142 779 mini_kdc.cc:121] Time spent starting KDC: real 1.062s user 0.002s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790394221564982-779-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790394221564982-779-0/krb5kdc/test-user.keytab.
Sep 26 03:44:09 dist-test-slave-n2th krb5kdc[3821](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394249, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:44:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:44:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.194.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.194.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.194.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790394221564982-779-0/krb5kdc/kudu_127.0.194.254.keytab.
Entry for principal kudu/127.0.194.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790394221564982-779-0/krb5kdc/kudu_127.0.194.254.keytab.
I20260926 03:44:09.674688 779 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
/tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790394221564982-779-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790394221564982-779-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790394221564982-779-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790394221564982-779-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.194.254:37081
--webserver_interface=127.0.194.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790394221564982-779-0/krb5kdc/kudu_127.0.194.254.keytab
--principal=kudu/127.0.194.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.194.212:36105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.194.254:37081
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790394221564982-779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790394221564982-779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790394221564982-779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:44:10.007422 3834 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:44:10.007761 3834 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:44:10.007951 3834 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:44:10.015403 3834 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:44:10.015547 3834 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:44:10.015654 3834 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:44:10.015754 3834 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:44:10.015841 3834 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:44:10.025852 3834 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.194.212:36105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790394221564982-779-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790394221564982-779-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.194.254:37081
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790394221564982-779-0/krb5kdc/kudu_127.0.194.254.keytab
--principal=kudu/127.0.194.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.194.254:37081
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790394221564982-779-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.194.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790394221564982-779-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:44:10.027238 3834 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:44:10.028522 3834 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:44:10.040536 3839 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:44:10.040448 3840 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:44:10.040505 3842 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:44:10.041464 3834 server_base.cc:1034] running on GCE node
I20260926 03:44:10.042279 3834 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:44:10.043880 3834 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:44:10.045121 3834 hybrid_clock.cc:648] HybridClock initialized: now 1790394250045084 us; error 58 us; skew 500 ppm
I20260926 03:44:10.045476 3834 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:44:10 dist-test-slave-n2th krb5kdc[3821](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394250, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:44:10.048674 3834 init.cc:377] Logged in from keytab as kudu/127.0.194.254@KRBTEST.COM (short username kudu)
I20260926 03:44:10.050203 3834 webserver.cc:460] Webserver started at http://127.0.194.254:39359/ using document root <none> and password file <none>
I20260926 03:44:10.050966 3834 fs_manager.cc:359] Metadata directory not provided
I20260926 03:44:10.051110 3834 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:44:10.051419 3834 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:44:10.054533 3834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790394221564982-779-0/minicluster-data/master-0/data/instance:
uuid: "e0a9aa5422b94c3d8e667db73f5a7256"
format_stamp: "Formatted at 2026-09-26 03:44:10 on dist-test-slave-n2th"
server_key: "87b0458ee026bbef02caed255c4ff837"
server_key_iv: "4e978816bcb7dc4b267461f50e81acb5"
server_key_version: "encryptionkey@0"
I20260926 03:44:10.055414 3834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790394221564982-779-0/minicluster-data/master-0/wal/instance:
uuid: "e0a9aa5422b94c3d8e667db73f5a7256"
format_stamp: "Formatted at 2026-09-26 03:44:10 on dist-test-slave-n2th"
server_key: "87b0458ee026bbef02caed255c4ff837"
server_key_iv: "4e978816bcb7dc4b267461f50e81acb5"
server_key_version: "encryptionkey@0"
I20260926 03:44:10.060940 3834 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 03:44:10.065292 3849 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:44:10.066715 3834 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260926 03:44:10.066982 3834 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790394221564982-779-0/minicluster-data/master-0/data,/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790394221564982-779-0/minicluster-data/master-0/wal
uuid: "e0a9aa5422b94c3d8e667db73f5a7256"
format_stamp: "Formatted at 2026-09-26 03:44:10 on dist-test-slave-n2th"
server_key: "87b0458ee026bbef02caed255c4ff837"
server_key_iv: "4e978816bcb7dc4b267461f50e81acb5"
server_key_version: "encryptionkey@0"
I20260926 03:44:10.067209 3834 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790394221564982-779-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790394221564982-779-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790394221564982-779-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:44:10.082849 3834 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:44:10.087642 3834 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:44:10.087970 3834 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:44:10.106573 3834 rpc_server.cc:307] RPC server started. Bound to: 127.0.194.254:37081
I20260926 03:44:10.106596 3900 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.194.254:37081 every 8 connection(s)
I20260926 03:44:10.108069 3834 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790394221564982-779-0/minicluster-data/master-0/data/info.pb
I20260926 03:44:10.111769 3901 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:44:10.114470 779 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu as pid 3834
I20260926 03:44:10.114712 779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790394221564982-779-0/minicluster-data/master-0/wal/instance
I20260926 03:44:10.115526 779 external_mini_cluster.cc:1428] Setting key ad9a6fa4ca0c91c528e0c70f7665d21d
I20260926 03:44:10.124830 3901 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0a9aa5422b94c3d8e667db73f5a7256: Bootstrap starting.
I20260926 03:44:10.128738 3901 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e0a9aa5422b94c3d8e667db73f5a7256: Neither blocks nor log segments found. Creating new log.
I20260926 03:44:10.130139 3901 log.cc:824] T 00000000000000000000000000000000 P e0a9aa5422b94c3d8e667db73f5a7256: Log is configured to *not* fsync() on all Append() calls
Sep 26 03:44:10 dist-test-slave-n2th krb5kdc[3821](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394249, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.194.254@KRBTEST.COM
I20260926 03:44:10.133494 3901 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0a9aa5422b94c3d8e667db73f5a7256: No bootstrap required, opened a new log
I20260926 03:44:10.139540 3904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:10.117700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57600 (local address 127.0.194.254:37081)
0926 03:44:10.118538 (+ 838us) server_negotiation.cc:206] Beginning negotiation
0926 03:44:10.118551 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 03:44:10.118596 (+ 45us) server_negotiation.cc:407] Connection header received
0926 03:44:10.120312 (+ 1716us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:44:10.120338 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:44:10.120773 (+ 435us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:44:10.121153 (+ 380us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:44:10.122917 (+ 1764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:10.124253 (+ 1336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:10.126007 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:10.126697 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:10.126916 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:10.131927 (+ 5011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:44:10.131936 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:44:10.131954 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:44:10.131992 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:44:10.135178 (+ 3186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:10.135954 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:10.135972 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:10.135979 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:10.136156 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:10.136573 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:10.136579 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:10.136582 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:10.137145 (+ 563us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:44:10.137507 (+ 362us) server_negotiation.cc:1035] Waiting for connection context
0926 03:44:10.137937 (+ 430us) server_negotiation.cc:299] Negotiation successful
0926 03:44:10.138392 (+ 455us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":624,"thread_start_us":409,"threads_started":1}
I20260926 03:44:10.141402 3901 raft_consensus.cc:348] T 00000000000000000000000000000000 P e0a9aa5422b94c3d8e667db73f5a7256 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0a9aa5422b94c3d8e667db73f5a7256" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 37081 } }
I20260926 03:44:10.141796 3901 raft_consensus.cc:374] T 00000000000000000000000000000000 P e0a9aa5422b94c3d8e667db73f5a7256 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:44:10.141916 3901 raft_consensus.cc:729] T 00000000000000000000000000000000 P e0a9aa5422b94c3d8e667db73f5a7256 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0a9aa5422b94c3d8e667db73f5a7256, State: Initialized, Role: FOLLOWER
I20260926 03:44:10.142500 3901 consensus_queue.cc:260] T 00000000000000000000000000000000 P e0a9aa5422b94c3d8e667db73f5a7256 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0a9aa5422b94c3d8e667db73f5a7256" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 37081 } }
I20260926 03:44:10.142819 3901 raft_consensus.cc:388] T 00000000000000000000000000000000 P e0a9aa5422b94c3d8e667db73f5a7256 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:44:10.142946 3901 raft_consensus.cc:482] T 00000000000000000000000000000000 P e0a9aa5422b94c3d8e667db73f5a7256 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:44:10.143103 3901 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e0a9aa5422b94c3d8e667db73f5a7256 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:44:10.145557 3901 raft_consensus.cc:504] T 00000000000000000000000000000000 P e0a9aa5422b94c3d8e667db73f5a7256 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0a9aa5422b94c3d8e667db73f5a7256" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 37081 } }
I20260926 03:44:10.146164 3901 leader_election.cc:302] T 00000000000000000000000000000000 P e0a9aa5422b94c3d8e667db73f5a7256 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e0a9aa5422b94c3d8e667db73f5a7256; no voters:
I20260926 03:44:10.146795 3901 leader_election.cc:288] T 00000000000000000000000000000000 P e0a9aa5422b94c3d8e667db73f5a7256 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:44:10.146948 3906 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e0a9aa5422b94c3d8e667db73f5a7256 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:44:10.147679 3906 raft_consensus.cc:686] T 00000000000000000000000000000000 P e0a9aa5422b94c3d8e667db73f5a7256 [term 1 LEADER]: Becoming Leader. State: Replica: e0a9aa5422b94c3d8e667db73f5a7256, State: Running, Role: LEADER
I20260926 03:44:10.148229 3906 consensus_queue.cc:237] T 00000000000000000000000000000000 P e0a9aa5422b94c3d8e667db73f5a7256 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0a9aa5422b94c3d8e667db73f5a7256" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 37081 } }
I20260926 03:44:10.148954 3901 sys_catalog.cc:564] T 00000000000000000000000000000000 P e0a9aa5422b94c3d8e667db73f5a7256 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:44:10.151350 3907 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0a9aa5422b94c3d8e667db73f5a7256 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e0a9aa5422b94c3d8e667db73f5a7256" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0a9aa5422b94c3d8e667db73f5a7256" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 37081 } } }
I20260926 03:44:10.151407 3908 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0a9aa5422b94c3d8e667db73f5a7256 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e0a9aa5422b94c3d8e667db73f5a7256. Latest consensus state: current_term: 1 leader_uuid: "e0a9aa5422b94c3d8e667db73f5a7256" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0a9aa5422b94c3d8e667db73f5a7256" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 37081 } } }
I20260926 03:44:10.151782 3907 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0a9aa5422b94c3d8e667db73f5a7256 [sys.catalog]: This master's current role is: LEADER
I20260926 03:44:10.151789 3908 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0a9aa5422b94c3d8e667db73f5a7256 [sys.catalog]: This master's current role is: LEADER
I20260926 03:44:10.152660 3915 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:44:10.161912 3915 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:44:10.170301 3915 catalog_manager.cc:1347] Generated new cluster ID: df95328c421f4d68a4eb9cef1d15c19c
I20260926 03:44:10.170444 3915 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:44:10.200959 3915 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:44:10.202332 3915 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:44:10.209599 3915 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e0a9aa5422b94c3d8e667db73f5a7256: Generated new TSK 0
I20260926 03:44:10.210467 3915 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.194.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.194.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.194.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790394221564982-779-0/krb5kdc/kudu_127.0.194.193.keytab.
Entry for principal kudu/127.0.194.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790394221564982-779-0/krb5kdc/kudu_127.0.194.193.keytab.
I20260926 03:44:10.266259 779 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
/tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790394221564982-779-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790394221564982-779-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790394221564982-779-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790394221564982-779-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.194.193:0
--local_ip_for_outbound_sockets=127.0.194.193
--webserver_interface=127.0.194.193
--webserver_port=0
--tserver_master_addrs=127.0.194.254:37081
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790394221564982-779-0/krb5kdc/kudu_127.0.194.193.keytab
--principal=kudu/127.0.194.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.194.212:36105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790394221564982-779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790394221564982-779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790394221564982-779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:44:10.597780 3927 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:44:10.598146 3927 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:44:10.598316 3927 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:44:10.605715 3927 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:44:10.605895 3927 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:44:10.606078 3927 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.194.193
I20260926 03:44:10.616114 3927 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.194.212:36105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790394221564982-779-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790394221564982-779-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790394221564982-779-0/krb5kdc/kudu_127.0.194.193.keytab
--principal=kudu/127.0.194.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.194.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790394221564982-779-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.194.193
--webserver_port=0
--tserver_master_addrs=127.0.194.254:37081
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.194.193
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790394221564982-779-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:44:10.617552 3927 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:44:10.618963 3927 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:44:10.632949 3932 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:44:10.632933 3933 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:44:10.633384 3935 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:44:10.633572 3927 server_base.cc:1034] running on GCE node
I20260926 03:44:10.634310 3927 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:44:10.635376 3927 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:44:10.636617 3927 hybrid_clock.cc:648] HybridClock initialized: now 1790394250636561 us; error 54 us; skew 500 ppm
I20260926 03:44:10.636958 3927 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:44:10 dist-test-slave-n2th krb5kdc[3821](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394250, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:44:10.640220 3927 init.cc:377] Logged in from keytab as kudu/127.0.194.193@KRBTEST.COM (short username kudu)
I20260926 03:44:10.641693 3927 webserver.cc:460] Webserver started at http://127.0.194.193:43801/ using document root <none> and password file <none>
I20260926 03:44:10.642400 3927 fs_manager.cc:359] Metadata directory not provided
I20260926 03:44:10.642545 3927 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:44:10.642911 3927 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:44:10.645874 3927 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790394221564982-779-0/minicluster-data/ts-0/data/instance:
uuid: "c1a5098f613444959f3798c4c2c44fa3"
format_stamp: "Formatted at 2026-09-26 03:44:10 on dist-test-slave-n2th"
server_key: "2f485a1302add4e643c29ecc55dac5d4"
server_key_iv: "5e4fbac40b063afb875f0899d505d758"
server_key_version: "encryptionkey@0"
I20260926 03:44:10.646688 3927 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790394221564982-779-0/minicluster-data/ts-0/wal/instance:
uuid: "c1a5098f613444959f3798c4c2c44fa3"
format_stamp: "Formatted at 2026-09-26 03:44:10 on dist-test-slave-n2th"
server_key: "2f485a1302add4e643c29ecc55dac5d4"
server_key_iv: "5e4fbac40b063afb875f0899d505d758"
server_key_version: "encryptionkey@0"
I20260926 03:44:10.651933 3927 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 03:44:10.655743 3942 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:44:10.657032 3927 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 03:44:10.657220 3927 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790394221564982-779-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790394221564982-779-0/minicluster-data/ts-0/wal
uuid: "c1a5098f613444959f3798c4c2c44fa3"
format_stamp: "Formatted at 2026-09-26 03:44:10 on dist-test-slave-n2th"
server_key: "2f485a1302add4e643c29ecc55dac5d4"
server_key_iv: "5e4fbac40b063afb875f0899d505d758"
server_key_version: "encryptionkey@0"
I20260926 03:44:10.657424 3927 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790394221564982-779-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790394221564982-779-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790394221564982-779-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:44:10.676743 3927 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:44:10.680886 3927 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:44:10.681258 3927 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:44:10.682289 3927 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:44:10.683751 3927 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:44:10.683857 3927 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:44:10.684011 3927 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:44:10.684083 3927 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:44:10.718782 3927 rpc_server.cc:307] RPC server started. Bound to: 127.0.194.193:39427
I20260926 03:44:10.718832 4054 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.194.193:39427 every 8 connection(s)
I20260926 03:44:10.720091 3927 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790394221564982-779-0/minicluster-data/ts-0/data/info.pb
I20260926 03:44:10.728048 779 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu as pid 3927
I20260926 03:44:10.728266 779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790394221564982-779-0/minicluster-data/ts-0/wal/instance
I20260926 03:44:10.728868 779 external_mini_cluster.cc:1428] Setting key 056270392887fecc69e8b4e67ff0effe
Sep 26 03:44:10 dist-test-slave-n2th krb5kdc[3821](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394250, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.193@KRBTEST.COM for kudu/127.0.194.254@KRBTEST.COM
I20260926 03:44:10.740777 4057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:10.722688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.193:45189 (local address 127.0.194.254:37081)
0926 03:44:10.723194 (+ 506us) server_negotiation.cc:206] Beginning negotiation
0926 03:44:10.723198 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:44:10.724081 (+ 883us) server_negotiation.cc:407] Connection header received
0926 03:44:10.725869 (+ 1788us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:44:10.725875 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:44:10.725965 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:44:10.726092 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:44:10.727690 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:10.728518 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:10.730261 (+ 1743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:10.730870 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:10.731002 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:10.734455 (+ 3453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:44:10.734465 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:44:10.734473 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:44:10.734521 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:44:10.737379 (+ 2858us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:10.738363 (+ 984us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:10.738367 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:10.738369 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:10.738483 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:10.738990 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:10.738994 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:10.738996 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:10.739264 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:44:10.739372 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0926 03:44:10.740327 (+ 955us) server_negotiation.cc:299] Negotiation successful
0926 03:44:10.740518 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":405,"thread_start_us":297,"threads_started":1}
I20260926 03:44:10.742340 4058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:10.723197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:37081 (local address 127.0.194.193:45189)
0926 03:44:10.723918 (+ 721us) negotiation.cc:107] Waiting for socket to connect
0926 03:44:10.723955 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 03:44:10.725608 (+ 1653us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:44:10.726318 (+ 710us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:44:10.726326 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:44:10.726931 (+ 605us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:44:10.727491 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:10.727508 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:10.728673 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:10.728678 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:10.730081 (+ 1403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:10.730094 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:10.731082 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:10.731087 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:10.731295 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:10.732023 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:44:10.732054 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:44:10.734203 (+ 2149us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:44:10.737533 (+ 3330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:44:10.737545 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:44:10.737567 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:44:10.738238 (+ 671us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:44:10.738592 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:44:10.738596 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:44:10.738600 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:44:10.738874 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:44:10.739398 (+ 524us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:44:10.739409 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:44:10.739884 (+ 475us) client_negotiation.cc:769] Sending connection context
0926 03:44:10.740200 (+ 316us) client_negotiation.cc:240] Negotiation successful
0926 03:44:10.740797 (+ 597us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":304,"threads_started":1}
I20260926 03:44:10.744114 4055 heartbeater.cc:347] Connected to a master server at 127.0.194.254:37081
I20260926 03:44:10.744546 4055 heartbeater.cc:464] Registering TS with master...
I20260926 03:44:10.745456 4055 heartbeater.cc:511] Master 127.0.194.254:37081 requested a full tablet report, sending...
I20260926 03:44:10.747885 3866 ts_manager.cc:194] Registered new tserver with Master: c1a5098f613444959f3798c4c2c44fa3 (127.0.194.193:39427)
WARNING: no policy specified for kudu/127.0.194.194@KRBTEST.COM; defaulting to no policy
I20260926 03:44:10.749424 3866 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.194.193@KRBTEST.COM'} at 127.0.194.193:45189
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.194.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.194.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790394221564982-779-0/krb5kdc/kudu_127.0.194.194.keytab.
Entry for principal kudu/127.0.194.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790394221564982-779-0/krb5kdc/kudu_127.0.194.194.keytab.
I20260926 03:44:10.776075 779 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
/tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790394221564982-779-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790394221564982-779-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790394221564982-779-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790394221564982-779-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.194.194:0
--local_ip_for_outbound_sockets=127.0.194.194
--webserver_interface=127.0.194.194
--webserver_port=0
--tserver_master_addrs=127.0.194.254:37081
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790394221564982-779-0/krb5kdc/kudu_127.0.194.194.keytab
--principal=kudu/127.0.194.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.194.212:36105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790394221564982-779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790394221564982-779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790394221564982-779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:44:11.109292 4061 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:44:11.109642 4061 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:44:11.109817 4061 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:44:11.117511 4061 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:44:11.117683 4061 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:44:11.117874 4061 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.194.194
I20260926 03:44:11.128221 4061 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.194.212:36105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790394221564982-779-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790394221564982-779-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790394221564982-779-0/krb5kdc/kudu_127.0.194.194.keytab
--principal=kudu/127.0.194.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.194.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790394221564982-779-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.194.194
--webserver_port=0
--tserver_master_addrs=127.0.194.254:37081
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.194.194
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790394221564982-779-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:44:11.129611 4061 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:44:11.130982 4061 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:44:11.144593 4069 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:44:11.144788 4066 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:44:11.144788 4067 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:44:11.145670 4061 server_base.cc:1034] running on GCE node
I20260926 03:44:11.146329 4061 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:44:11.147480 4061 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:44:11.148701 4061 hybrid_clock.cc:648] HybridClock initialized: now 1790394251148630 us; error 66 us; skew 500 ppm
I20260926 03:44:11.149091 4061 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:44:11 dist-test-slave-n2th krb5kdc[3821](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394251, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:44:11.152973 4061 init.cc:377] Logged in from keytab as kudu/127.0.194.194@KRBTEST.COM (short username kudu)
I20260926 03:44:11.154567 4061 webserver.cc:460] Webserver started at http://127.0.194.194:38379/ using document root <none> and password file <none>
I20260926 03:44:11.155323 4061 fs_manager.cc:359] Metadata directory not provided
I20260926 03:44:11.155459 4061 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:44:11.155777 4061 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:44:11.158799 4061 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790394221564982-779-0/minicluster-data/ts-1/data/instance:
uuid: "b2fe74525ba044a0bfb9e6953101fede"
format_stamp: "Formatted at 2026-09-26 03:44:11 on dist-test-slave-n2th"
server_key: "4968e8222d4f05f349d551e08854c3c7"
server_key_iv: "121153c904a9b9ba80bf15e7f726f252"
server_key_version: "encryptionkey@0"
I20260926 03:44:11.159617 4061 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790394221564982-779-0/minicluster-data/ts-1/wal/instance:
uuid: "b2fe74525ba044a0bfb9e6953101fede"
format_stamp: "Formatted at 2026-09-26 03:44:11 on dist-test-slave-n2th"
server_key: "4968e8222d4f05f349d551e08854c3c7"
server_key_iv: "121153c904a9b9ba80bf15e7f726f252"
server_key_version: "encryptionkey@0"
I20260926 03:44:11.165194 4061 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 03:44:11.168926 4076 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:44:11.170492 4061 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 03:44:11.170671 4061 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790394221564982-779-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790394221564982-779-0/minicluster-data/ts-1/wal
uuid: "b2fe74525ba044a0bfb9e6953101fede"
format_stamp: "Formatted at 2026-09-26 03:44:11 on dist-test-slave-n2th"
server_key: "4968e8222d4f05f349d551e08854c3c7"
server_key_iv: "121153c904a9b9ba80bf15e7f726f252"
server_key_version: "encryptionkey@0"
I20260926 03:44:11.170940 4061 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790394221564982-779-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790394221564982-779-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790394221564982-779-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:44:11.196388 4061 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:44:11.201082 4061 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:44:11.201419 4061 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:44:11.202425 4061 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:44:11.203841 4061 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:44:11.203914 4061 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:44:11.204062 4061 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:44:11.204120 4061 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:44:11.237591 4061 rpc_server.cc:307] RPC server started. Bound to: 127.0.194.194:39167
I20260926 03:44:11.237684 4188 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.194.194:39167 every 8 connection(s)
I20260926 03:44:11.239017 4061 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790394221564982-779-0/minicluster-data/ts-1/data/info.pb
I20260926 03:44:11.246502 779 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu as pid 4061
I20260926 03:44:11.246698 779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790394221564982-779-0/minicluster-data/ts-1/wal/instance
I20260926 03:44:11.247280 779 external_mini_cluster.cc:1428] Setting key 6342c20807652fd963ff7bcaa27ee9ed
Sep 26 03:44:11 dist-test-slave-n2th krb5kdc[3821](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394251, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.194@KRBTEST.COM for kudu/127.0.194.254@KRBTEST.COM
I20260926 03:44:11.261950 4191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:11.241728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.194:49967 (local address 127.0.194.254:37081)
0926 03:44:11.242186 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0926 03:44:11.242191 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:44:11.243327 (+ 1136us) server_negotiation.cc:407] Connection header received
0926 03:44:11.245161 (+ 1834us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:44:11.245165 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:44:11.245233 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:44:11.245344 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:44:11.247431 (+ 2087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:11.248351 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:11.250829 (+ 2478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:11.251246 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:11.251359 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:11.255775 (+ 4416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:44:11.255781 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:44:11.255784 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:44:11.255816 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:44:11.258512 (+ 2696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:11.259406 (+ 894us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:11.259410 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:11.259413 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:11.259547 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:11.260011 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:11.260020 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:11.260023 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:11.260293 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:44:11.260456 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0926 03:44:11.261458 (+ 1002us) server_negotiation.cc:299] Negotiation successful
0926 03:44:11.261650 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":238,"threads_started":1}
I20260926 03:44:11.263592 4192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:11.242288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:37081 (local address 127.0.194.194:49967)
0926 03:44:11.243110 (+ 822us) negotiation.cc:107] Waiting for socket to connect
0926 03:44:11.243169 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0926 03:44:11.244887 (+ 1718us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:44:11.245594 (+ 707us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:44:11.245603 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:44:11.246286 (+ 683us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:44:11.247209 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:11.247234 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:11.248512 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:11.248516 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:11.250633 (+ 2117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:11.250653 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:11.251505 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:11.251515 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:11.251883 (+ 368us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:11.252657 (+ 774us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:44:11.252701 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:44:11.255487 (+ 2786us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:44:11.258688 (+ 3201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:44:11.258697 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:44:11.258717 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:44:11.259255 (+ 538us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:44:11.259669 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:44:11.259674 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:44:11.259678 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:44:11.259885 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:44:11.260485 (+ 600us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:44:11.260497 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:44:11.260969 (+ 472us) client_negotiation.cc:769] Sending connection context
0926 03:44:11.261344 (+ 375us) client_negotiation.cc:240] Negotiation successful
0926 03:44:11.261987 (+ 643us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":566,"thread_start_us":278,"threads_started":1}
I20260926 03:44:11.265142 4189 heartbeater.cc:347] Connected to a master server at 127.0.194.254:37081
I20260926 03:44:11.265475 4189 heartbeater.cc:464] Registering TS with master...
I20260926 03:44:11.266222 4189 heartbeater.cc:511] Master 127.0.194.254:37081 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.194.195@KRBTEST.COM; defaulting to no policy
I20260926 03:44:11.267796 3866 ts_manager.cc:194] Registered new tserver with Master: b2fe74525ba044a0bfb9e6953101fede (127.0.194.194:39167)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.194.195@KRBTEST.COM" created.
I20260926 03:44:11.268813 3866 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.194.194@KRBTEST.COM'} at 127.0.194.194:49967
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.194.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790394221564982-779-0/krb5kdc/kudu_127.0.194.195.keytab.
Entry for principal kudu/127.0.194.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790394221564982-779-0/krb5kdc/kudu_127.0.194.195.keytab.
I20260926 03:44:11.292044 779 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
/tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790394221564982-779-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790394221564982-779-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790394221564982-779-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790394221564982-779-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.194.195:0
--local_ip_for_outbound_sockets=127.0.194.195
--webserver_interface=127.0.194.195
--webserver_port=0
--tserver_master_addrs=127.0.194.254:37081
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790394221564982-779-0/krb5kdc/kudu_127.0.194.195.keytab
--principal=kudu/127.0.194.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.194.212:36105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790394221564982-779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790394221564982-779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790394221564982-779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:44:11.617718 4195 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:44:11.618018 4195 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:44:11.618113 4195 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:44:11.625487 4195 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:44:11.625612 4195 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:44:11.625815 4195 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.194.195
I20260926 03:44:11.636066 4195 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.194.212:36105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790394221564982-779-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790394221564982-779-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790394221564982-779-0/krb5kdc/kudu_127.0.194.195.keytab
--principal=kudu/127.0.194.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.194.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790394221564982-779-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.194.195
--webserver_port=0
--tserver_master_addrs=127.0.194.254:37081
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.194.195
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790394221564982-779-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:44:11.637403 4195 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:44:11.638881 4195 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:44:11.652711 4200 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:44:11.652647 4201 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:44:11.652599 4203 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:44:11.653447 4195 server_base.cc:1034] running on GCE node
I20260926 03:44:11.654040 4195 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:44:11.655020 4195 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:44:11.656229 4195 hybrid_clock.cc:648] HybridClock initialized: now 1790394251656163 us; error 62 us; skew 500 ppm
I20260926 03:44:11.656567 4195 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:44:11 dist-test-slave-n2th krb5kdc[3821](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394251, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:44:11.660081 4195 init.cc:377] Logged in from keytab as kudu/127.0.194.195@KRBTEST.COM (short username kudu)
I20260926 03:44:11.661708 4195 webserver.cc:460] Webserver started at http://127.0.194.195:34097/ using document root <none> and password file <none>
I20260926 03:44:11.662474 4195 fs_manager.cc:359] Metadata directory not provided
I20260926 03:44:11.662593 4195 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:44:11.662920 4195 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:44:11.665891 4195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790394221564982-779-0/minicluster-data/ts-2/data/instance:
uuid: "1a963b08bac140dba980b639505ddb05"
format_stamp: "Formatted at 2026-09-26 03:44:11 on dist-test-slave-n2th"
server_key: "90f5b87cd5e04c073e90318e86abb77b"
server_key_iv: "b4ee8903b8880ea1abf41ddd082acc22"
server_key_version: "encryptionkey@0"
I20260926 03:44:11.666683 4195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790394221564982-779-0/minicluster-data/ts-2/wal/instance:
uuid: "1a963b08bac140dba980b639505ddb05"
format_stamp: "Formatted at 2026-09-26 03:44:11 on dist-test-slave-n2th"
server_key: "90f5b87cd5e04c073e90318e86abb77b"
server_key_iv: "b4ee8903b8880ea1abf41ddd082acc22"
server_key_version: "encryptionkey@0"
I20260926 03:44:11.672369 4195 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260926 03:44:11.676147 4210 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:44:11.677592 4195 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 03:44:11.677794 4195 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790394221564982-779-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790394221564982-779-0/minicluster-data/ts-2/wal
uuid: "1a963b08bac140dba980b639505ddb05"
format_stamp: "Formatted at 2026-09-26 03:44:11 on dist-test-slave-n2th"
server_key: "90f5b87cd5e04c073e90318e86abb77b"
server_key_iv: "b4ee8903b8880ea1abf41ddd082acc22"
server_key_version: "encryptionkey@0"
I20260926 03:44:11.678002 4195 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790394221564982-779-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790394221564982-779-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790394221564982-779-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:44:11.704023 4195 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:44:11.708750 4195 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:44:11.709087 4195 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:44:11.710100 4195 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:44:11.711501 4195 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:44:11.711572 4195 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:44:11.711748 4195 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:44:11.711825 4195 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:44:11.748844 4195 rpc_server.cc:307] RPC server started. Bound to: 127.0.194.195:44349
I20260926 03:44:11.748898 4322 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.194.195:44349 every 8 connection(s)
I20260926 03:44:11.750280 4195 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790394221564982-779-0/minicluster-data/ts-2/data/info.pb
I20260926 03:44:11.751926 779 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu as pid 4195
I20260926 03:44:11.752115 779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790394221564982-779-0/minicluster-data/ts-2/wal/instance
I20260926 03:44:11.752665 779 external_mini_cluster.cc:1428] Setting key badf9256ffca662d14ba1ba4ac819d51
I20260926 03:44:11.753151 4055 heartbeater.cc:503] Master 127.0.194.254:37081 was elected leader, sending a full tablet report...
Sep 26 03:44:11 dist-test-slave-n2th krb5kdc[3821](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394251, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.195@KRBTEST.COM for kudu/127.0.194.254@KRBTEST.COM
I20260926 03:44:11.771976 4191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:11.753306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.195:34823 (local address 127.0.194.254:37081)
0926 03:44:11.753585 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0926 03:44:11.753591 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:44:11.755184 (+ 1593us) server_negotiation.cc:407] Connection header received
0926 03:44:11.757187 (+ 2003us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:44:11.757191 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:44:11.757252 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:44:11.757345 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:44:11.758836 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:11.759585 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:11.761206 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:11.761688 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:11.761787 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:11.765178 (+ 3391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:44:11.765183 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:44:11.765186 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:44:11.765219 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:44:11.768927 (+ 3708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:11.769699 (+ 772us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:11.769703 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:11.769706 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:11.769828 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:11.770258 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:11.770261 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:11.770264 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:11.770563 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:44:11.770672 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0926 03:44:11.771467 (+ 795us) server_negotiation.cc:299] Negotiation successful
0926 03:44:11.771676 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143}
I20260926 03:44:11.772989 4325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:11.753746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:37081 (local address 127.0.194.195:34823)
0926 03:44:11.755024 (+ 1278us) negotiation.cc:107] Waiting for socket to connect
0926 03:44:11.755076 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 03:44:11.756937 (+ 1861us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:44:11.757550 (+ 613us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:44:11.757558 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:44:11.758058 (+ 500us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:44:11.758644 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:11.758661 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:11.759719 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:11.759723 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:11.761079 (+ 1356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:11.761095 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:11.761882 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:11.761887 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:11.762083 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:11.762770 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:44:11.762803 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:44:11.764979 (+ 2176us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:44:11.769071 (+ 4092us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:44:11.769081 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:44:11.769103 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:44:11.769574 (+ 471us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:44:11.769923 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:44:11.769926 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:44:11.769928 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:44:11.770158 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:44:11.770671 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:44:11.770677 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:44:11.771096 (+ 419us) client_negotiation.cc:769] Sending connection context
0926 03:44:11.771363 (+ 267us) client_negotiation.cc:240] Negotiation successful
0926 03:44:11.771839 (+ 476us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":903,"thread_start_us":661,"threads_started":1}
I20260926 03:44:11.774257 4323 heartbeater.cc:347] Connected to a master server at 127.0.194.254:37081
I20260926 03:44:11.774569 4323 heartbeater.cc:464] Registering TS with master...
I20260926 03:44:11.775379 4323 heartbeater.cc:511] Master 127.0.194.254:37081 requested a full tablet report, sending...
I20260926 03:44:11.776859 3865 ts_manager.cc:194] Registered new tserver with Master: 1a963b08bac140dba980b639505ddb05 (127.0.194.195:44349)
I20260926 03:44:11.777686 3865 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.194.195@KRBTEST.COM'} at 127.0.194.195:34823
I20260926 03:44:11.780997 779 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 03:44:11 dist-test-slave-n2th krb5kdc[3821](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394251, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 26 03:44:11 dist-test-slave-n2th krb5kdc[3821](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394251, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.194.254@KRBTEST.COM
I20260926 03:44:11.830889 4191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:11.816715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57606 (local address 127.0.194.254:37081)
0926 03:44:11.816892 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0926 03:44:11.816896 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:44:11.817329 (+ 433us) server_negotiation.cc:407] Connection header received
0926 03:44:11.817502 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:44:11.817505 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:44:11.817570 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:44:11.817667 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:44:11.819092 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:11.819854 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:11.821054 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:11.821443 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:11.821536 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:11.825805 (+ 4269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:44:11.825810 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:44:11.825814 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:44:11.825850 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:44:11.828399 (+ 2549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:11.829213 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:11.829217 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:11.829220 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:11.829344 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:11.829667 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:11.829670 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:11.829672 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:11.829983 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:44:11.830167 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0926 03:44:11.830397 (+ 230us) server_negotiation.cc:299] Negotiation successful
0926 03:44:11.830558 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
W20260926 03:44:11.831514 3865 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:57606
I20260926 03:44:11.833541 779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu pid 3927
Sep 26 03:44:11 dist-test-slave-n2th krb5kdc[3821](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394251, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.194.193@KRBTEST.COM
I20260926 03:44:11.851730 4334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:11.834532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33870 (local address 127.0.194.193:39427)
0926 03:44:11.835179 (+ 647us) server_negotiation.cc:206] Beginning negotiation
0926 03:44:11.835190 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 03:44:11.835257 (+ 67us) server_negotiation.cc:407] Connection header received
0926 03:44:11.835327 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:44:11.835333 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:44:11.835562 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:44:11.835743 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:44:11.837221 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:11.838250 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:11.839678 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:11.840264 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:11.840414 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:11.844865 (+ 4451us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:44:11.844880 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:44:11.844896 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:44:11.844941 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:44:11.848317 (+ 3376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:11.849420 (+ 1103us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:11.849428 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:11.849430 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:11.849579 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:11.850037 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:11.850041 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:11.850044 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:11.850410 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:44:11.850737 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0926 03:44:11.851078 (+ 341us) server_negotiation.cc:299] Negotiation successful
0926 03:44:11.851367 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":474,"thread_start_us":299,"threads_started":1}
W20260926 03:44:11.853070 4050 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:33870
W20260926 03:44:11.854193 779 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu pid 3927: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 03:44:11.854385 779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu with pid 3927
I20260926 03:44:11.870833 779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu pid 4061
Sep 26 03:44:11 dist-test-slave-n2th krb5kdc[3821](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394251, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.194.194@KRBTEST.COM
I20260926 03:44:11.886996 4335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:11.871740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50858 (local address 127.0.194.194:39167)
0926 03:44:11.872383 (+ 643us) server_negotiation.cc:206] Beginning negotiation
0926 03:44:11.872391 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:44:11.872418 (+ 27us) server_negotiation.cc:407] Connection header received
0926 03:44:11.872478 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:44:11.872485 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:44:11.872699 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:44:11.872838 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:44:11.874077 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:11.875224 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:11.876554 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:11.877101 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:11.877199 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:11.881286 (+ 4087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:44:11.881297 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:44:11.881314 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:44:11.881348 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:44:11.884425 (+ 3077us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:11.885036 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:11.885043 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:11.885045 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:11.885156 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:11.885513 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:11.885518 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:11.885520 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:11.885900 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:44:11.886139 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0926 03:44:11.886351 (+ 212us) server_negotiation.cc:299] Negotiation successful
0926 03:44:11.886549 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":493,"thread_start_us":347,"threads_started":1}
W20260926 03:44:11.888309 4184 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:50858
W20260926 03:44:11.889387 779 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu pid 4061: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 03:44:11.889530 779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu with pid 4061
I20260926 03:44:11.905066 779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu pid 4195
Sep 26 03:44:11 dist-test-slave-n2th krb5kdc[3821](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394251, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.194.195@KRBTEST.COM
I20260926 03:44:11.921938 4336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:11.906176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34650 (local address 127.0.194.195:44349)
0926 03:44:11.906974 (+ 798us) server_negotiation.cc:206] Beginning negotiation
0926 03:44:11.906980 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:44:11.907011 (+ 31us) server_negotiation.cc:407] Connection header received
0926 03:44:11.907069 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:44:11.907074 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:44:11.907301 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:44:11.907429 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:44:11.908778 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:11.909878 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:11.911245 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:11.911804 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:11.911908 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:11.916684 (+ 4776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:44:11.916698 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:44:11.916713 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:44:11.916750 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:44:11.919531 (+ 2781us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:11.920187 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:11.920194 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:11.920196 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:11.920324 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:11.920634 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:11.920637 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:11.920639 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:11.920970 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:44:11.921195 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0926 03:44:11.921449 (+ 254us) server_negotiation.cc:299] Negotiation successful
0926 03:44:11.921644 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":642,"thread_start_us":424,"threads_started":1}
W20260926 03:44:11.923333 4318 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:34650
W20260926 03:44:11.924431 779 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu pid 4195: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 03:44:11.924592 779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu with pid 4195
I20260926 03:44:11.939733 779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu pid 3834
I20260926 03:44:11.940965 3896 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:44:12.044651 779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu with pid 3834
2026-09-26T03:44:12Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3674 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790394221564982-779-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:44:12 dist-test-slave-n2th krb5kdc[4340](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:44:12 dist-test-slave-n2th krb5kdc[4340](info): set up 2 sockets
Sep 26 03:44:12 dist-test-slave-n2th krb5kdc[4340](info): commencing operation
krb5kdc: starting...
W20260926 03:44:13.159226 779 mini_kdc.cc:121] Time spent starting KDC: real 1.072s user 0.006s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790394221564982-779-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790394221564982-779-0/krb5kdc/test-user.keytab.
Sep 26 03:44:13 dist-test-slave-n2th krb5kdc[4340](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394253, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:44:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:44:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.194.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.194.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.194.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790394221564982-779-0/krb5kdc/kudu_127.0.194.254.keytab.
Entry for principal kudu/127.0.194.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790394221564982-779-0/krb5kdc/kudu_127.0.194.254.keytab.
I20260926 03:44:13.360174 779 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
/tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790394221564982-779-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790394221564982-779-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790394221564982-779-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790394221564982-779-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.194.254:35573
--webserver_interface=127.0.194.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790394221564982-779-0/krb5kdc/kudu_127.0.194.254.keytab
--principal=kudu/127.0.194.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.194.212:46147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.194.254:35573
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790394221564982-779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790394221564982-779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790394221564982-779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:44:13.690428 4353 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:44:13.690790 4353 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:44:13.690918 4353 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:44:13.698222 4353 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:44:13.698335 4353 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:44:13.698381 4353 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:44:13.698441 4353 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:44:13.698484 4353 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:44:13.708722 4353 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.194.212:46147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790394221564982-779-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790394221564982-779-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.194.254:35573
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790394221564982-779-0/krb5kdc/kudu_127.0.194.254.keytab
--principal=kudu/127.0.194.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.194.254:35573
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790394221564982-779-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.194.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790394221564982-779-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:44:13.710052 4353 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:44:13.711404 4353 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:44:13.725174 4358 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:44:13.725467 4359 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:44:13.726001 4361 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:44:13.726099 4353 server_base.cc:1034] running on GCE node
I20260926 03:44:13.727074 4353 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:44:13.728605 4353 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:44:13.729835 4353 hybrid_clock.cc:648] HybridClock initialized: now 1790394253729784 us; error 39 us; skew 500 ppm
I20260926 03:44:13.730204 4353 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:44:13 dist-test-slave-n2th krb5kdc[4340](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394253, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:44:13.737098 4353 init.cc:377] Logged in from keytab as kudu/127.0.194.254@KRBTEST.COM (short username kudu)
I20260926 03:44:13.738899 4353 webserver.cc:460] Webserver started at http://127.0.194.254:39343/ using document root <none> and password file <none>
I20260926 03:44:13.739653 4353 fs_manager.cc:359] Metadata directory not provided
I20260926 03:44:13.739799 4353 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:44:13.740145 4353 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:44:13.743342 4353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790394221564982-779-0/minicluster-data/master-0/data/instance:
uuid: "5b89a31f67844fcba42a7b7dec5b26c1"
format_stamp: "Formatted at 2026-09-26 03:44:13 on dist-test-slave-n2th"
server_key: "99b497733a0416dd8d60764bd3c6236c"
server_key_iv: "b73cf2006240017fdd90084719874f23"
server_key_version: "encryptionkey@0"
I20260926 03:44:13.744184 4353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790394221564982-779-0/minicluster-data/master-0/wal/instance:
uuid: "5b89a31f67844fcba42a7b7dec5b26c1"
format_stamp: "Formatted at 2026-09-26 03:44:13 on dist-test-slave-n2th"
server_key: "99b497733a0416dd8d60764bd3c6236c"
server_key_iv: "b73cf2006240017fdd90084719874f23"
server_key_version: "encryptionkey@0"
I20260926 03:44:13.749570 4353 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 03:44:13.753126 4368 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:44:13.754482 4353 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 03:44:13.754673 4353 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790394221564982-779-0/minicluster-data/master-0/data,/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790394221564982-779-0/minicluster-data/master-0/wal
uuid: "5b89a31f67844fcba42a7b7dec5b26c1"
format_stamp: "Formatted at 2026-09-26 03:44:13 on dist-test-slave-n2th"
server_key: "99b497733a0416dd8d60764bd3c6236c"
server_key_iv: "b73cf2006240017fdd90084719874f23"
server_key_version: "encryptionkey@0"
I20260926 03:44:13.754968 4353 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790394221564982-779-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790394221564982-779-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790394221564982-779-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:44:13.770327 4353 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:44:13.775048 4353 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:44:13.775374 4353 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:44:13.794620 4353 rpc_server.cc:307] RPC server started. Bound to: 127.0.194.254:35573
I20260926 03:44:13.794729 4419 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.194.254:35573 every 8 connection(s)
I20260926 03:44:13.796130 4353 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790394221564982-779-0/minicluster-data/master-0/data/info.pb
I20260926 03:44:13.799965 4420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:44:13.800171 779 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu as pid 4353
I20260926 03:44:13.800422 779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790394221564982-779-0/minicluster-data/master-0/wal/instance
I20260926 03:44:13.801065 779 external_mini_cluster.cc:1428] Setting key b39ebd59102e3cf7a74a5c61f9ec0946
I20260926 03:44:13.815333 4420 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b89a31f67844fcba42a7b7dec5b26c1: Bootstrap starting.
Sep 26 03:44:13 dist-test-slave-n2th krb5kdc[4340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394253, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.194.254@KRBTEST.COM
I20260926 03:44:13.819226 4420 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5b89a31f67844fcba42a7b7dec5b26c1: Neither blocks nor log segments found. Creating new log.
I20260926 03:44:13.820740 4420 log.cc:824] T 00000000000000000000000000000000 P 5b89a31f67844fcba42a7b7dec5b26c1: Log is configured to *not* fsync() on all Append() calls
I20260926 03:44:13.824187 4420 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b89a31f67844fcba42a7b7dec5b26c1: No bootstrap required, opened a new log
I20260926 03:44:13.826817 4423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:13.803231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42694 (local address 127.0.194.254:35573)
0926 03:44:13.804068 (+ 837us) server_negotiation.cc:206] Beginning negotiation
0926 03:44:13.804083 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 03:44:13.804137 (+ 54us) server_negotiation.cc:407] Connection header received
0926 03:44:13.806936 (+ 2799us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:44:13.806969 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:44:13.807542 (+ 573us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:44:13.808042 (+ 500us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:44:13.809395 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:13.811438 (+ 2043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:13.812780 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:13.813836 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:13.814021 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:13.818303 (+ 4282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:44:13.818314 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:44:13.818333 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:44:13.818391 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:44:13.822158 (+ 3767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:13.822933 (+ 775us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:13.822942 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:13.822950 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:13.823158 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:13.823554 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:13.823563 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:13.823566 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:13.824282 (+ 716us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:44:13.824555 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0926 03:44:13.824977 (+ 422us) server_negotiation.cc:299] Negotiation successful
0926 03:44:13.825419 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":585,"thread_start_us":353,"threads_started":1}
I20260926 03:44:13.832527 4420 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5b89a31f67844fcba42a7b7dec5b26c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b89a31f67844fcba42a7b7dec5b26c1" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 35573 } }
I20260926 03:44:13.832906 4420 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5b89a31f67844fcba42a7b7dec5b26c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:44:13.832976 4420 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5b89a31f67844fcba42a7b7dec5b26c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a31f67844fcba42a7b7dec5b26c1, State: Initialized, Role: FOLLOWER
I20260926 03:44:13.833552 4420 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5b89a31f67844fcba42a7b7dec5b26c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b89a31f67844fcba42a7b7dec5b26c1" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 35573 } }
I20260926 03:44:13.833771 4420 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5b89a31f67844fcba42a7b7dec5b26c1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:44:13.833864 4420 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5b89a31f67844fcba42a7b7dec5b26c1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:44:13.834044 4420 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5b89a31f67844fcba42a7b7dec5b26c1 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:44:13.836700 4420 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5b89a31f67844fcba42a7b7dec5b26c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b89a31f67844fcba42a7b7dec5b26c1" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 35573 } }
I20260926 03:44:13.837168 4420 leader_election.cc:302] T 00000000000000000000000000000000 P 5b89a31f67844fcba42a7b7dec5b26c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5b89a31f67844fcba42a7b7dec5b26c1; no voters:
I20260926 03:44:13.837702 4420 leader_election.cc:288] T 00000000000000000000000000000000 P 5b89a31f67844fcba42a7b7dec5b26c1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:44:13.837886 4425 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5b89a31f67844fcba42a7b7dec5b26c1 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:44:13.838841 4425 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5b89a31f67844fcba42a7b7dec5b26c1 [term 1 LEADER]: Becoming Leader. State: Replica: 5b89a31f67844fcba42a7b7dec5b26c1, State: Running, Role: LEADER
I20260926 03:44:13.839423 4425 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5b89a31f67844fcba42a7b7dec5b26c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b89a31f67844fcba42a7b7dec5b26c1" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 35573 } }
I20260926 03:44:13.840282 4420 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5b89a31f67844fcba42a7b7dec5b26c1 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:44:13.842175 4427 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b89a31f67844fcba42a7b7dec5b26c1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5b89a31f67844fcba42a7b7dec5b26c1. Latest consensus state: current_term: 1 leader_uuid: "5b89a31f67844fcba42a7b7dec5b26c1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b89a31f67844fcba42a7b7dec5b26c1" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 35573 } } }
I20260926 03:44:13.842592 4427 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b89a31f67844fcba42a7b7dec5b26c1 [sys.catalog]: This master's current role is: LEADER
I20260926 03:44:13.842362 4426 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b89a31f67844fcba42a7b7dec5b26c1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5b89a31f67844fcba42a7b7dec5b26c1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b89a31f67844fcba42a7b7dec5b26c1" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 35573 } } }
I20260926 03:44:13.842796 4426 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b89a31f67844fcba42a7b7dec5b26c1 [sys.catalog]: This master's current role is: LEADER
I20260926 03:44:13.846567 4430 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:44:13.852382 4430 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:44:13.862072 4430 catalog_manager.cc:1347] Generated new cluster ID: 2d5845f53732423c8beb59bff5d52761
I20260926 03:44:13.862221 4430 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:44:13.872720 4430 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:44:13.874034 4430 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:44:13.888583 4430 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5b89a31f67844fcba42a7b7dec5b26c1: Generated new TSK 0
I20260926 03:44:13.889675 4430 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.194.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.194.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.194.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790394221564982-779-0/krb5kdc/kudu_127.0.194.193.keytab.
Entry for principal kudu/127.0.194.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790394221564982-779-0/krb5kdc/kudu_127.0.194.193.keytab.
I20260926 03:44:13.940436 779 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
/tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790394221564982-779-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790394221564982-779-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790394221564982-779-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790394221564982-779-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.194.193:0
--local_ip_for_outbound_sockets=127.0.194.193
--webserver_interface=127.0.194.193
--webserver_port=0
--tserver_master_addrs=127.0.194.254:35573
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790394221564982-779-0/krb5kdc/kudu_127.0.194.193.keytab
--principal=kudu/127.0.194.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.194.212:46147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790394221564982-779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790394221564982-779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790394221564982-779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:44:14.269444 4446 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:44:14.269761 4446 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:44:14.269861 4446 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:44:14.277293 4446 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:44:14.277428 4446 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:44:14.277542 4446 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.194.193
I20260926 03:44:14.287977 4446 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.194.212:46147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790394221564982-779-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790394221564982-779-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790394221564982-779-0/krb5kdc/kudu_127.0.194.193.keytab
--principal=kudu/127.0.194.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.194.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790394221564982-779-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.194.193
--webserver_port=0
--tserver_master_addrs=127.0.194.254:35573
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.194.193
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790394221564982-779-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:44:14.289494 4446 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:44:14.290931 4446 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:44:14.304077 4452 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:44:14.304080 4451 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:44:14.305176 4454 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:44:14.306465 4446 server_base.cc:1034] running on GCE node
I20260926 03:44:14.307243 4446 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:44:14.308398 4446 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:44:14.309680 4446 hybrid_clock.cc:648] HybridClock initialized: now 1790394254309624 us; error 62 us; skew 500 ppm
I20260926 03:44:14.310081 4446 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:44:14 dist-test-slave-n2th krb5kdc[4340](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394254, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:44:14.313658 4446 init.cc:377] Logged in from keytab as kudu/127.0.194.193@KRBTEST.COM (short username kudu)
I20260926 03:44:14.315440 4446 webserver.cc:460] Webserver started at http://127.0.194.193:37239/ using document root <none> and password file <none>
I20260926 03:44:14.316177 4446 fs_manager.cc:359] Metadata directory not provided
I20260926 03:44:14.316331 4446 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:44:14.316653 4446 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:44:14.320782 4446 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790394221564982-779-0/minicluster-data/ts-0/data/instance:
uuid: "2918675534cd4f249bc33c268d744ef1"
format_stamp: "Formatted at 2026-09-26 03:44:14 on dist-test-slave-n2th"
server_key: "c0ae1dfcb516c440bc81986180ff8ffb"
server_key_iv: "389aa61a8b9cabaf7ef3636ccf42d085"
server_key_version: "encryptionkey@0"
I20260926 03:44:14.321707 4446 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790394221564982-779-0/minicluster-data/ts-0/wal/instance:
uuid: "2918675534cd4f249bc33c268d744ef1"
format_stamp: "Formatted at 2026-09-26 03:44:14 on dist-test-slave-n2th"
server_key: "c0ae1dfcb516c440bc81986180ff8ffb"
server_key_iv: "389aa61a8b9cabaf7ef3636ccf42d085"
server_key_version: "encryptionkey@0"
I20260926 03:44:14.328141 4446 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260926 03:44:14.332170 4461 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:44:14.333734 4446 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 03:44:14.333961 4446 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790394221564982-779-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790394221564982-779-0/minicluster-data/ts-0/wal
uuid: "2918675534cd4f249bc33c268d744ef1"
format_stamp: "Formatted at 2026-09-26 03:44:14 on dist-test-slave-n2th"
server_key: "c0ae1dfcb516c440bc81986180ff8ffb"
server_key_iv: "389aa61a8b9cabaf7ef3636ccf42d085"
server_key_version: "encryptionkey@0"
I20260926 03:44:14.334203 4446 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790394221564982-779-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790394221564982-779-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790394221564982-779-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:44:14.354466 4446 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:44:14.359355 4446 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:44:14.359654 4446 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:44:14.360680 4446 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:44:14.362056 4446 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:44:14.362167 4446 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:44:14.362323 4446 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:44:14.362398 4446 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:44:14.396051 4446 rpc_server.cc:307] RPC server started. Bound to: 127.0.194.193:34731
I20260926 03:44:14.396095 4573 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.194.193:34731 every 8 connection(s)
I20260926 03:44:14.397348 4446 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790394221564982-779-0/minicluster-data/ts-0/data/info.pb
I20260926 03:44:14.400826 779 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu as pid 4446
I20260926 03:44:14.401054 779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790394221564982-779-0/minicluster-data/ts-0/wal/instance
I20260926 03:44:14.401753 779 external_mini_cluster.cc:1428] Setting key ea8437d69f3cee6a96abb24baad5a5d1
Sep 26 03:44:14 dist-test-slave-n2th krb5kdc[4340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394254, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.193@KRBTEST.COM for kudu/127.0.194.254@KRBTEST.COM
I20260926 03:44:14.419132 4576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:14.399913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.193:54371 (local address 127.0.194.254:35573)
0926 03:44:14.400357 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0926 03:44:14.400361 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:44:14.401386 (+ 1025us) server_negotiation.cc:407] Connection header received
0926 03:44:14.403326 (+ 1940us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:44:14.403331 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:44:14.403442 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:44:14.403567 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:44:14.405104 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:14.405828 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:14.407521 (+ 1693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:14.407936 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:14.408093 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:14.411960 (+ 3867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:44:14.411969 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:44:14.411976 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:44:14.412023 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:44:14.415676 (+ 3653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:14.416438 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:14.416447 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:14.416453 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:14.416601 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:14.417120 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:14.417128 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:14.417132 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:14.417535 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:44:14.417707 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0926 03:44:14.418423 (+ 716us) server_negotiation.cc:299] Negotiation successful
0926 03:44:14.418685 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":251,"threads_started":1}
I20260926 03:44:14.419878 4577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:14.400392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:35573 (local address 127.0.194.193:54371)
0926 03:44:14.401209 (+ 817us) negotiation.cc:107] Waiting for socket to connect
0926 03:44:14.401253 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 03:44:14.403052 (+ 1799us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:44:14.403834 (+ 782us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:44:14.403843 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:44:14.404345 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:44:14.404928 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:14.404946 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:14.405992 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:14.405997 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:14.407362 (+ 1365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:14.407374 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:14.408205 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:14.408213 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:14.408460 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:14.409210 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:44:14.409237 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:44:14.411681 (+ 2444us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:44:14.415828 (+ 4147us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:44:14.415835 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:44:14.415850 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:44:14.416285 (+ 435us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:44:14.416735 (+ 450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:44:14.416743 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:44:14.416747 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:44:14.416981 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:44:14.417667 (+ 686us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:44:14.417673 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:44:14.418010 (+ 337us) client_negotiation.cc:769] Sending connection context
0926 03:44:14.418238 (+ 228us) client_negotiation.cc:240] Negotiation successful
0926 03:44:14.418652 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":545,"thread_start_us":349,"threads_started":1}
WARNING: no policy specified for kudu/127.0.194.194@KRBTEST.COM; defaulting to no policy
I20260926 03:44:14.421558 4574 heartbeater.cc:347] Connected to a master server at 127.0.194.254:35573
I20260926 03:44:14.421875 4574 heartbeater.cc:464] Registering TS with master...
I20260926 03:44:14.422653 4574 heartbeater.cc:511] Master 127.0.194.254:35573 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.194.194@KRBTEST.COM" created.
I20260926 03:44:14.424852 4385 ts_manager.cc:194] Registered new tserver with Master: 2918675534cd4f249bc33c268d744ef1 (127.0.194.193:34731)
I20260926 03:44:14.426266 4385 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.194.193@KRBTEST.COM'} at 127.0.194.193:54371
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.194.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790394221564982-779-0/krb5kdc/kudu_127.0.194.194.keytab.
Entry for principal kudu/127.0.194.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790394221564982-779-0/krb5kdc/kudu_127.0.194.194.keytab.
I20260926 03:44:14.448045 779 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
/tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790394221564982-779-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790394221564982-779-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790394221564982-779-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790394221564982-779-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.194.194:0
--local_ip_for_outbound_sockets=127.0.194.194
--webserver_interface=127.0.194.194
--webserver_port=0
--tserver_master_addrs=127.0.194.254:35573
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790394221564982-779-0/krb5kdc/kudu_127.0.194.194.keytab
--principal=kudu/127.0.194.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.194.212:46147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790394221564982-779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790394221564982-779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790394221564982-779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:44:14.768028 4580 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:44:14.768308 4580 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:44:14.768527 4580 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:44:14.775682 4580 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:44:14.775800 4580 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:44:14.776036 4580 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.194.194
I20260926 03:44:14.787506 4580 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.194.212:46147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790394221564982-779-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790394221564982-779-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790394221564982-779-0/krb5kdc/kudu_127.0.194.194.keytab
--principal=kudu/127.0.194.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.194.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790394221564982-779-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.194.194
--webserver_port=0
--tserver_master_addrs=127.0.194.254:35573
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.194.194
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790394221564982-779-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:44:14.788848 4580 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:44:14.790287 4580 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:44:14.803898 4588 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:44:14.803963 4586 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:44:14.804129 4585 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:44:14.805299 4580 server_base.cc:1034] running on GCE node
I20260926 03:44:14.806010 4580 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:44:14.807073 4580 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:44:14.808318 4580 hybrid_clock.cc:648] HybridClock initialized: now 1790394254808268 us; error 43 us; skew 500 ppm
I20260926 03:44:14.808650 4580 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:44:14 dist-test-slave-n2th krb5kdc[4340](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394254, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:44:14.811939 4580 init.cc:377] Logged in from keytab as kudu/127.0.194.194@KRBTEST.COM (short username kudu)
I20260926 03:44:14.813500 4580 webserver.cc:460] Webserver started at http://127.0.194.194:39519/ using document root <none> and password file <none>
I20260926 03:44:14.814235 4580 fs_manager.cc:359] Metadata directory not provided
I20260926 03:44:14.814380 4580 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:44:14.814703 4580 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:44:14.817754 4580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790394221564982-779-0/minicluster-data/ts-1/data/instance:
uuid: "f558d9f67f9e499dad558e5ea190e199"
format_stamp: "Formatted at 2026-09-26 03:44:14 on dist-test-slave-n2th"
server_key: "1112631bfbec7d7104d5dab28fac1828"
server_key_iv: "9b445ef4b40612c2f402ccdf2ef33120"
server_key_version: "encryptionkey@0"
I20260926 03:44:14.818552 4580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790394221564982-779-0/minicluster-data/ts-1/wal/instance:
uuid: "f558d9f67f9e499dad558e5ea190e199"
format_stamp: "Formatted at 2026-09-26 03:44:14 on dist-test-slave-n2th"
server_key: "1112631bfbec7d7104d5dab28fac1828"
server_key_iv: "9b445ef4b40612c2f402ccdf2ef33120"
server_key_version: "encryptionkey@0"
I20260926 03:44:14.823729 4580 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.003s
I20260926 03:44:14.827363 4595 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:44:14.828634 4580 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260926 03:44:14.828811 4580 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790394221564982-779-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790394221564982-779-0/minicluster-data/ts-1/wal
uuid: "f558d9f67f9e499dad558e5ea190e199"
format_stamp: "Formatted at 2026-09-26 03:44:14 on dist-test-slave-n2th"
server_key: "1112631bfbec7d7104d5dab28fac1828"
server_key_iv: "9b445ef4b40612c2f402ccdf2ef33120"
server_key_version: "encryptionkey@0"
I20260926 03:44:14.829046 4580 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790394221564982-779-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790394221564982-779-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790394221564982-779-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:44:14.859279 4580 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:44:14.863911 4580 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:44:14.864214 4580 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:44:14.865185 4580 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:44:14.866528 4580 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:44:14.866655 4580 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:44:14.866822 4580 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:44:14.866878 4580 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:44:14.899927 4580 rpc_server.cc:307] RPC server started. Bound to: 127.0.194.194:41287
I20260926 03:44:14.900020 4707 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.194.194:41287 every 8 connection(s)
I20260926 03:44:14.901260 4580 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790394221564982-779-0/minicluster-data/ts-1/data/info.pb
I20260926 03:44:14.907641 779 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu as pid 4580
I20260926 03:44:14.907855 779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790394221564982-779-0/minicluster-data/ts-1/wal/instance
I20260926 03:44:14.908450 779 external_mini_cluster.cc:1428] Setting key 3b384931d1c6575b2efff098a5863202
Sep 26 03:44:14 dist-test-slave-n2th krb5kdc[4340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394254, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.194@KRBTEST.COM for kudu/127.0.194.254@KRBTEST.COM
I20260926 03:44:14.929206 4576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:14.903913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.194:52131 (local address 127.0.194.254:35573)
0926 03:44:14.904100 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0926 03:44:14.904104 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:44:14.905216 (+ 1112us) server_negotiation.cc:407] Connection header received
0926 03:44:14.906996 (+ 1780us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:44:14.907000 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:44:14.907071 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:44:14.907160 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:44:14.909041 (+ 1881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:14.910140 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:14.913106 (+ 2966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:14.914933 (+ 1827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:14.915114 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:14.920002 (+ 4888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:44:14.920010 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:44:14.920015 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:44:14.920062 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:44:14.924926 (+ 4864us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:14.926014 (+ 1088us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:14.926023 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:14.926029 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:14.926206 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:14.926820 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:14.926828 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:14.926834 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:14.927226 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:44:14.927361 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0926 03:44:14.928464 (+ 1103us) server_negotiation.cc:299] Negotiation successful
0926 03:44:14.928738 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 03:44:14.931758 4710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:14.904371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:35573 (local address 127.0.194.194:52131)
0926 03:44:14.905031 (+ 660us) negotiation.cc:107] Waiting for socket to connect
0926 03:44:14.905071 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 03:44:14.906663 (+ 1592us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:44:14.907401 (+ 738us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:44:14.907409 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:44:14.907997 (+ 588us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:44:14.908824 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:14.908852 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:14.910365 (+ 1513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:14.910373 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:14.912869 (+ 2496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:14.912885 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:14.915222 (+ 2337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:14.915233 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:14.915589 (+ 356us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:14.916463 (+ 874us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:44:14.916498 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:44:14.919676 (+ 3178us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:44:14.925129 (+ 5453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:44:14.925142 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:44:14.925165 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:44:14.925855 (+ 690us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:44:14.926371 (+ 516us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:44:14.926378 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:44:14.926384 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:44:14.926648 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:44:14.927470 (+ 822us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:44:14.927481 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:44:14.927990 (+ 509us) client_negotiation.cc:769] Sending connection context
0926 03:44:14.929447 (+ 1457us) client_negotiation.cc:240] Negotiation successful
0926 03:44:14.930084 (+ 637us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":274,"threads_started":1}
I20260926 03:44:14.933645 4708 heartbeater.cc:347] Connected to a master server at 127.0.194.254:35573
I20260926 03:44:14.934074 4708 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.194.195@KRBTEST.COM; defaulting to no policy
I20260926 03:44:14.935072 4708 heartbeater.cc:511] Master 127.0.194.254:35573 requested a full tablet report, sending...
I20260926 03:44:14.936829 4385 ts_manager.cc:194] Registered new tserver with Master: f558d9f67f9e499dad558e5ea190e199 (127.0.194.194:41287)
I20260926 03:44:14.938220 4385 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.194.194@KRBTEST.COM'} at 127.0.194.194:52131
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.194.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.194.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790394221564982-779-0/krb5kdc/kudu_127.0.194.195.keytab.
Entry for principal kudu/127.0.194.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790394221564982-779-0/krb5kdc/kudu_127.0.194.195.keytab.
I20260926 03:44:14.966207 779 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
/tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790394221564982-779-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790394221564982-779-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790394221564982-779-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790394221564982-779-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.194.195:0
--local_ip_for_outbound_sockets=127.0.194.195
--webserver_interface=127.0.194.195
--webserver_port=0
--tserver_master_addrs=127.0.194.254:35573
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790394221564982-779-0/krb5kdc/kudu_127.0.194.195.keytab
--principal=kudu/127.0.194.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.194.212:46147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790394221564982-779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790394221564982-779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790394221564982-779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:44:15.292487 4713 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:44:15.292778 4713 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:44:15.292881 4713 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:44:15.300596 4713 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:44:15.300712 4713 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:44:15.300824 4713 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.194.195
I20260926 03:44:15.311295 4713 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.194.212:46147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790394221564982-779-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790394221564982-779-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790394221564982-779-0/krb5kdc/kudu_127.0.194.195.keytab
--principal=kudu/127.0.194.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.194.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790394221564982-779-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.194.195
--webserver_port=0
--tserver_master_addrs=127.0.194.254:35573
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.194.195
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790394221564982-779-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:44:15.312582 4713 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:44:15.313825 4713 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:44:15.327148 4718 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:44:15.327184 4721 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:44:15.327111 4719 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:44:15.328339 4713 server_base.cc:1034] running on GCE node
I20260926 03:44:15.329008 4713 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:44:15.329913 4713 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:44:15.331152 4713 hybrid_clock.cc:648] HybridClock initialized: now 1790394255331097 us; error 51 us; skew 500 ppm
I20260926 03:44:15.331492 4713 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:44:15 dist-test-slave-n2th krb5kdc[4340](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394255, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:44:15.334867 4713 init.cc:377] Logged in from keytab as kudu/127.0.194.195@KRBTEST.COM (short username kudu)
I20260926 03:44:15.336263 4713 webserver.cc:460] Webserver started at http://127.0.194.195:34333/ using document root <none> and password file <none>
I20260926 03:44:15.336910 4713 fs_manager.cc:359] Metadata directory not provided
I20260926 03:44:15.337006 4713 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:44:15.337255 4713 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:44:15.340561 4713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790394221564982-779-0/minicluster-data/ts-2/data/instance:
uuid: "c263798d07154b5e8f13be22a90c16a1"
format_stamp: "Formatted at 2026-09-26 03:44:15 on dist-test-slave-n2th"
server_key: "b1b65282f753f628b7d9fc1e8845369e"
server_key_iv: "8b33a4e8b28f6b06ee0c44737afc40ad"
server_key_version: "encryptionkey@0"
I20260926 03:44:15.341282 4713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790394221564982-779-0/minicluster-data/ts-2/wal/instance:
uuid: "c263798d07154b5e8f13be22a90c16a1"
format_stamp: "Formatted at 2026-09-26 03:44:15 on dist-test-slave-n2th"
server_key: "b1b65282f753f628b7d9fc1e8845369e"
server_key_iv: "8b33a4e8b28f6b06ee0c44737afc40ad"
server_key_version: "encryptionkey@0"
I20260926 03:44:15.346395 4713 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260926 03:44:15.349849 4728 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:44:15.351281 4713 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 03:44:15.351487 4713 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790394221564982-779-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790394221564982-779-0/minicluster-data/ts-2/wal
uuid: "c263798d07154b5e8f13be22a90c16a1"
format_stamp: "Formatted at 2026-09-26 03:44:15 on dist-test-slave-n2th"
server_key: "b1b65282f753f628b7d9fc1e8845369e"
server_key_iv: "8b33a4e8b28f6b06ee0c44737afc40ad"
server_key_version: "encryptionkey@0"
I20260926 03:44:15.351688 4713 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790394221564982-779-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790394221564982-779-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790394221564982-779-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:44:15.375866 4713 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:44:15.380407 4713 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:44:15.380710 4713 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:44:15.381685 4713 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:44:15.383028 4713 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:44:15.383100 4713 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:44:15.383255 4713 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:44:15.383314 4713 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:44:15.419209 4713 rpc_server.cc:307] RPC server started. Bound to: 127.0.194.195:40101
I20260926 03:44:15.419253 4840 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.194.195:40101 every 8 connection(s)
I20260926 03:44:15.420497 4713 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790394221564982-779-0/minicluster-data/ts-2/data/info.pb
I20260926 03:44:15.429558 4574 heartbeater.cc:503] Master 127.0.194.254:35573 was elected leader, sending a full tablet report...
I20260926 03:44:15.430861 779 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu as pid 4713
I20260926 03:44:15.431063 779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790394221564982-779-0/minicluster-data/ts-2/wal/instance
I20260926 03:44:15.431562 779 external_mini_cluster.cc:1428] Setting key 9b9c78a8dd79dc029df3d634a26f1cb4
Sep 26 03:44:15 dist-test-slave-n2th krb5kdc[4340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394255, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.195@KRBTEST.COM for kudu/127.0.194.254@KRBTEST.COM
I20260926 03:44:15.440953 4576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:15.422883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.195:39781 (local address 127.0.194.254:35573)
0926 03:44:15.423105 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0926 03:44:15.423112 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:44:15.424379 (+ 1267us) server_negotiation.cc:407] Connection header received
0926 03:44:15.426100 (+ 1721us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:44:15.426104 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:44:15.426187 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:44:15.426286 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:44:15.427838 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:15.428611 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:15.430220 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:15.430857 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:15.430983 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:15.435209 (+ 4226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:44:15.435217 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:44:15.435222 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:44:15.435267 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:44:15.438070 (+ 2803us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:15.438867 (+ 797us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:15.438871 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:15.438874 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:15.438998 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:15.439388 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:15.439391 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:15.439393 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:15.439651 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:44:15.439761 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0926 03:44:15.440542 (+ 781us) server_negotiation.cc:299] Negotiation successful
0926 03:44:15.440711 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260926 03:44:15.442099 4843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:15.423473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:35573 (local address 127.0.194.195:39781)
0926 03:44:15.424198 (+ 725us) negotiation.cc:107] Waiting for socket to connect
0926 03:44:15.424236 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 03:44:15.425823 (+ 1587us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:44:15.426498 (+ 675us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:44:15.426506 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:44:15.427084 (+ 578us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:44:15.427653 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:15.427670 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:15.428743 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:15.428746 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:15.430071 (+ 1325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:15.430086 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:15.431099 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:15.431106 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:15.431401 (+ 295us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:15.432114 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:44:15.432151 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:44:15.434895 (+ 2744us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:44:15.438191 (+ 3296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:44:15.438198 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:44:15.438254 (+ 56us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:44:15.438694 (+ 440us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:44:15.439092 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:44:15.439095 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:44:15.439098 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:44:15.439284 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:44:15.439762 (+ 478us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:44:15.439768 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:44:15.440153 (+ 385us) client_negotiation.cc:769] Sending connection context
0926 03:44:15.440387 (+ 234us) client_negotiation.cc:240] Negotiation successful
0926 03:44:15.440903 (+ 516us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":309,"threads_started":1}
I20260926 03:44:15.443508 4841 heartbeater.cc:347] Connected to a master server at 127.0.194.254:35573
I20260926 03:44:15.443794 4841 heartbeater.cc:464] Registering TS with master...
I20260926 03:44:15.444453 4841 heartbeater.cc:511] Master 127.0.194.254:35573 requested a full tablet report, sending...
I20260926 03:44:15.445842 4385 ts_manager.cc:194] Registered new tserver with Master: c263798d07154b5e8f13be22a90c16a1 (127.0.194.195:40101)
I20260926 03:44:15.446950 4385 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.194.195@KRBTEST.COM'} at 127.0.194.195:39781
I20260926 03:44:15.447795 779 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 03:44:15 dist-test-slave-n2th krb5kdc[4340](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394255, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 03:44:15 dist-test-slave-n2th krb5kdc[4340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394255, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.194.193@KRBTEST.COM
I20260926 03:44:15.495396 4848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:15.480558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33130 (local address 127.0.194.193:34731)
0926 03:44:15.481198 (+ 640us) server_negotiation.cc:206] Beginning negotiation
0926 03:44:15.481210 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 03:44:15.481250 (+ 40us) server_negotiation.cc:407] Connection header received
0926 03:44:15.481306 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:44:15.481312 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:44:15.481533 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:44:15.481709 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:44:15.483131 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:15.484142 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:15.485385 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:15.486018 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:15.486160 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:15.490159 (+ 3999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:44:15.490170 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:44:15.490185 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:44:15.490220 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:44:15.493058 (+ 2838us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:15.493670 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:15.493677 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:15.493679 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:15.493791 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:15.494098 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:15.494101 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:15.494102 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:15.494412 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:44:15.494632 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0926 03:44:15.494914 (+ 282us) server_negotiation.cc:299] Negotiation successful
0926 03:44:15.495109 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":467,"thread_start_us":309,"threads_started":1}
Sep 26 03:44:15 dist-test-slave-n2th krb5kdc[4340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394255, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.194.254@KRBTEST.COM
I20260926 03:44:15.513131 4576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:15.499590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42698 (local address 127.0.194.254:35573)
0926 03:44:15.499821 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0926 03:44:15.499827 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:44:15.500047 (+ 220us) server_negotiation.cc:407] Connection header received
0926 03:44:15.500244 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:44:15.500250 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:44:15.500323 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:44:15.500431 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:44:15.501626 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:15.502341 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:15.503654 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:15.504075 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:15.504206 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:15.508067 (+ 3861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:44:15.508078 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:44:15.508081 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:44:15.508114 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:44:15.510544 (+ 2430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:15.511210 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:15.511216 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:15.511220 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:15.511347 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:15.511717 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:15.511722 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:15.511725 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:15.512031 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:44:15.512275 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0926 03:44:15.512595 (+ 320us) server_negotiation.cc:299] Negotiation successful
0926 03:44:15.512803 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
W20260926 03:44:15.513680 4384 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42698
I20260926 03:44:15.514700 779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu pid 4446
I20260926 03:44:15.527329 4848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:15.515731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33136 (local address 127.0.194.193:34731)
0926 03:44:15.515937 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0926 03:44:15.515942 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:44:15.516096 (+ 154us) server_negotiation.cc:407] Connection header received
0926 03:44:15.516292 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:44:15.516301 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:44:15.516388 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:44:15.516510 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:44:15.517662 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:15.518337 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:15.519590 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:15.520021 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:15.520156 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:15.522469 (+ 2313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:44:15.522474 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:44:15.522477 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:44:15.522504 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:44:15.524876 (+ 2372us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:15.525488 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:15.525496 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:15.525499 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:15.525621 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:15.525987 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:15.525993 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:15.525997 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:15.526235 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:44:15.526487 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0926 03:44:15.526809 (+ 322us) server_negotiation.cc:299] Negotiation successful
0926 03:44:15.527024 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 03:44:15.527974 4569 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:44:15.611048 779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu with pid 4446
I20260926 03:44:15.628017 779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu pid 4580
Sep 26 03:44:15 dist-test-slave-n2th krb5kdc[4340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394255, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.194.194@KRBTEST.COM
I20260926 03:44:15.645234 4853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:15.629071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53958 (local address 127.0.194.194:41287)
0926 03:44:15.629617 (+ 546us) server_negotiation.cc:206] Beginning negotiation
0926 03:44:15.629624 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:44:15.629650 (+ 26us) server_negotiation.cc:407] Connection header received
0926 03:44:15.629708 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:44:15.629715 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:44:15.629947 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:44:15.630077 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:44:15.631302 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:15.632312 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:15.633798 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:15.634350 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:15.634446 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:15.639097 (+ 4651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:44:15.639109 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:44:15.639125 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:44:15.639162 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:44:15.642230 (+ 3068us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:15.643033 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:15.643040 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:15.643043 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:15.643163 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:15.643563 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:15.643573 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:15.643575 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:15.643935 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:44:15.644248 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0926 03:44:15.644586 (+ 338us) server_negotiation.cc:299] Negotiation successful
0926 03:44:15.644858 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":409,"thread_start_us":295,"threads_started":1}
I20260926 03:44:15.646606 4703 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:44:15.732736 779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu with pid 4580
I20260926 03:44:15.749570 779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu pid 4713
Sep 26 03:44:15 dist-test-slave-n2th krb5kdc[4340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394255, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.194.195@KRBTEST.COM
I20260926 03:44:15.765689 4855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:15.750515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43876 (local address 127.0.194.195:40101)
0926 03:44:15.751169 (+ 654us) server_negotiation.cc:206] Beginning negotiation
0926 03:44:15.751176 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:44:15.751204 (+ 28us) server_negotiation.cc:407] Connection header received
0926 03:44:15.751261 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:44:15.751268 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:44:15.751488 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:44:15.751624 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:44:15.752878 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:15.753919 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:15.755313 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:15.755922 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:15.756019 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:15.760405 (+ 4386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:44:15.760417 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:44:15.760433 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:44:15.760470 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:44:15.763388 (+ 2918us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:15.764027 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:15.764034 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:15.764036 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:15.764149 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:15.764456 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:15.764459 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:15.764461 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:15.764777 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:44:15.764994 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0926 03:44:15.765208 (+ 214us) server_negotiation.cc:299] Negotiation successful
0926 03:44:15.765410 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":502,"thread_start_us":350,"threads_started":1}
I20260926 03:44:15.766994 4836 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:44:15.851469 779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu with pid 4713
I20260926 03:44:15.867187 779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu pid 4353
I20260926 03:44:15.868468 4415 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:44:15.971092 779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu with pid 4353
2026-09-26T03:44:15Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (3926 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790394221564982-779-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:44:16 dist-test-slave-n2th krb5kdc[4860](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:44:16 dist-test-slave-n2th krb5kdc[4860](info): set up 2 sockets
Sep 26 03:44:16 dist-test-slave-n2th krb5kdc[4860](info): commencing operation
krb5kdc: starting...
W20260926 03:44:17.063319 779 mini_kdc.cc:121] Time spent starting KDC: real 1.050s user 0.000s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790394221564982-779-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790394221564982-779-0/krb5kdc/test-user.keytab.
Sep 26 03:44:17 dist-test-slave-n2th krb5kdc[4860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394257, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:44:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:44:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.194.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.194.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.194.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790394221564982-779-0/krb5kdc/kudu_127.0.194.254.keytab.
Entry for principal kudu/127.0.194.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790394221564982-779-0/krb5kdc/kudu_127.0.194.254.keytab.
I20260926 03:44:17.271236 779 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
/tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790394221564982-779-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790394221564982-779-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790394221564982-779-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790394221564982-779-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.194.254:42827
--webserver_interface=127.0.194.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790394221564982-779-0/krb5kdc/kudu_127.0.194.254.keytab
--principal=kudu/127.0.194.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.194.212:35045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.194.254:42827
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790394221564982-779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790394221564982-779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790394221564982-779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:44:17.594889 4873 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:44:17.595184 4873 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:44:17.595275 4873 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:44:17.602653 4873 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:44:17.602846 4873 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:44:17.602913 4873 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:44:17.602968 4873 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:44:17.603009 4873 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:44:17.612984 4873 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.194.212:35045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790394221564982-779-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790394221564982-779-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.194.254:42827
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790394221564982-779-0/krb5kdc/kudu_127.0.194.254.keytab
--principal=kudu/127.0.194.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.194.254:42827
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790394221564982-779-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.194.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790394221564982-779-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:44:17.614236 4873 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:44:17.615520 4873 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:44:17.627337 4878 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:44:17.627312 4879 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:44:17.627409 4881 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:44:17.628250 4873 server_base.cc:1034] running on GCE node
I20260926 03:44:17.629119 4873 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:44:17.630625 4873 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:44:17.631896 4873 hybrid_clock.cc:648] HybridClock initialized: now 1790394257631848 us; error 43 us; skew 500 ppm
I20260926 03:44:17.632251 4873 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:44:17 dist-test-slave-n2th krb5kdc[4860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394257, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:44:17.635396 4873 init.cc:377] Logged in from keytab as kudu/127.0.194.254@KRBTEST.COM (short username kudu)
I20260926 03:44:17.636598 4873 fs_manager.cc:359] Metadata directory not provided
I20260926 03:44:17.636687 4873 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:44:17.636960 4873 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:44:17.640126 4873 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790394221564982-779-0/minicluster-data/master-0/data/instance:
uuid: "0416f884577f4295b3ca5e95f6f101be"
format_stamp: "Formatted at 2026-09-26 03:44:17 on dist-test-slave-n2th"
server_key: "3c12ad93ece3a6f62eeaa8d78ba478ab"
server_key_iv: "2be92f8a04abc8407ae6650f9fd9b065"
server_key_version: "encryptionkey@0"
I20260926 03:44:17.640897 4873 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790394221564982-779-0/minicluster-data/master-0/wal/instance:
uuid: "0416f884577f4295b3ca5e95f6f101be"
format_stamp: "Formatted at 2026-09-26 03:44:17 on dist-test-slave-n2th"
server_key: "3c12ad93ece3a6f62eeaa8d78ba478ab"
server_key_iv: "2be92f8a04abc8407ae6650f9fd9b065"
server_key_version: "encryptionkey@0"
I20260926 03:44:17.646024 4873 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 03:44:17.649670 4887 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:44:17.651153 4873 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260926 03:44:17.651305 4873 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790394221564982-779-0/minicluster-data/master-0/data,/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790394221564982-779-0/minicluster-data/master-0/wal
uuid: "0416f884577f4295b3ca5e95f6f101be"
format_stamp: "Formatted at 2026-09-26 03:44:17 on dist-test-slave-n2th"
server_key: "3c12ad93ece3a6f62eeaa8d78ba478ab"
server_key_iv: "2be92f8a04abc8407ae6650f9fd9b065"
server_key_version: "encryptionkey@0"
I20260926 03:44:17.651497 4873 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790394221564982-779-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790394221564982-779-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790394221564982-779-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:44:17.669474 4873 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:44:17.673743 4873 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:44:17.673985 4873 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:44:17.692670 4873 rpc_server.cc:307] RPC server started. Bound to: 127.0.194.254:42827
I20260926 03:44:17.692688 4938 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.194.254:42827 every 8 connection(s)
I20260926 03:44:17.693601 4873 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790394221564982-779-0/minicluster-data/master-0/data/info.pb
I20260926 03:44:17.697492 4939 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:44:17.701385 779 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu as pid 4873
I20260926 03:44:17.701622 779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790394221564982-779-0/minicluster-data/master-0/wal/instance
I20260926 03:44:17.702412 779 external_mini_cluster.cc:1428] Setting key 163887b9c6c98cdc04c082fda18e5281
I20260926 03:44:17.709348 4939 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0416f884577f4295b3ca5e95f6f101be: Bootstrap starting.
I20260926 03:44:17.713155 4939 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0416f884577f4295b3ca5e95f6f101be: Neither blocks nor log segments found. Creating new log.
I20260926 03:44:17.714622 4939 log.cc:824] T 00000000000000000000000000000000 P 0416f884577f4295b3ca5e95f6f101be: Log is configured to *not* fsync() on all Append() calls
I20260926 03:44:17.718341 4939 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0416f884577f4295b3ca5e95f6f101be: No bootstrap required, opened a new log
Sep 26 03:44:17 dist-test-slave-n2th krb5kdc[4860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394257, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.194.254@KRBTEST.COM
I20260926 03:44:17.728186 4939 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0416f884577f4295b3ca5e95f6f101be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0416f884577f4295b3ca5e95f6f101be" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 42827 } }
I20260926 03:44:17.728576 4939 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0416f884577f4295b3ca5e95f6f101be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:44:17.728615 4942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:17.704580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33806 (local address 127.0.194.254:42827)
0926 03:44:17.705415 (+ 835us) server_negotiation.cc:206] Beginning negotiation
0926 03:44:17.705427 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 03:44:17.705466 (+ 39us) server_negotiation.cc:407] Connection header received
0926 03:44:17.707763 (+ 2297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:44:17.707786 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:44:17.708180 (+ 394us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:44:17.708502 (+ 322us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:44:17.710344 (+ 1842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:17.711769 (+ 1425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:17.713998 (+ 2229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:17.715297 (+ 1299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:17.716153 (+ 856us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:17.720706 (+ 4553us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:44:17.720716 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:44:17.720736 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:44:17.720803 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:44:17.724423 (+ 3620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:17.725274 (+ 851us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:17.725281 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:17.725287 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:17.725449 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:17.725830 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:17.725833 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:17.725835 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:17.726378 (+ 543us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:44:17.726591 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0926 03:44:17.727099 (+ 508us) server_negotiation.cc:299] Negotiation successful
0926 03:44:17.727494 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":627,"thread_start_us":409,"threads_started":1}
I20260926 03:44:17.728706 4939 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0416f884577f4295b3ca5e95f6f101be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0416f884577f4295b3ca5e95f6f101be, State: Initialized, Role: FOLLOWER
I20260926 03:44:17.729336 4939 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0416f884577f4295b3ca5e95f6f101be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0416f884577f4295b3ca5e95f6f101be" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 42827 } }
I20260926 03:44:17.729592 4939 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0416f884577f4295b3ca5e95f6f101be [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:44:17.729741 4939 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0416f884577f4295b3ca5e95f6f101be [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:44:17.729928 4939 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0416f884577f4295b3ca5e95f6f101be [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:44:17.732834 4939 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0416f884577f4295b3ca5e95f6f101be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0416f884577f4295b3ca5e95f6f101be" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 42827 } }
I20260926 03:44:17.733263 4939 leader_election.cc:302] T 00000000000000000000000000000000 P 0416f884577f4295b3ca5e95f6f101be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0416f884577f4295b3ca5e95f6f101be; no voters:
I20260926 03:44:17.733770 4939 leader_election.cc:288] T 00000000000000000000000000000000 P 0416f884577f4295b3ca5e95f6f101be [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:44:17.733924 4944 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0416f884577f4295b3ca5e95f6f101be [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:44:17.734597 4944 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0416f884577f4295b3ca5e95f6f101be [term 1 LEADER]: Becoming Leader. State: Replica: 0416f884577f4295b3ca5e95f6f101be, State: Running, Role: LEADER
I20260926 03:44:17.735121 4944 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0416f884577f4295b3ca5e95f6f101be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0416f884577f4295b3ca5e95f6f101be" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 42827 } }
I20260926 03:44:17.735857 4939 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0416f884577f4295b3ca5e95f6f101be [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:44:17.738221 4945 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0416f884577f4295b3ca5e95f6f101be [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0416f884577f4295b3ca5e95f6f101be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0416f884577f4295b3ca5e95f6f101be" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 42827 } } }
I20260926 03:44:17.738801 4945 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0416f884577f4295b3ca5e95f6f101be [sys.catalog]: This master's current role is: LEADER
I20260926 03:44:17.739868 4953 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:44:17.739689 4946 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0416f884577f4295b3ca5e95f6f101be [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0416f884577f4295b3ca5e95f6f101be. Latest consensus state: current_term: 1 leader_uuid: "0416f884577f4295b3ca5e95f6f101be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0416f884577f4295b3ca5e95f6f101be" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 42827 } } }
I20260926 03:44:17.740080 4946 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0416f884577f4295b3ca5e95f6f101be [sys.catalog]: This master's current role is: LEADER
I20260926 03:44:17.746361 4953 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:44:17.756820 4953 catalog_manager.cc:1347] Generated new cluster ID: bee522434bea440aa2f7b3fd2ce24686
I20260926 03:44:17.756955 4953 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:44:17.771030 4953 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:44:17.772436 4953 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:44:17.785007 4953 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0416f884577f4295b3ca5e95f6f101be: Generated new TSK 0
I20260926 03:44:17.786196 4953 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.194.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.194.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.194.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790394221564982-779-0/krb5kdc/kudu_127.0.194.193.keytab.
Entry for principal kudu/127.0.194.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790394221564982-779-0/krb5kdc/kudu_127.0.194.193.keytab.
I20260926 03:44:17.848958 779 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
/tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790394221564982-779-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790394221564982-779-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790394221564982-779-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790394221564982-779-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.194.193:0
--local_ip_for_outbound_sockets=127.0.194.193
--webserver_interface=127.0.194.193
--webserver_port=0
--tserver_master_addrs=127.0.194.254:42827
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790394221564982-779-0/krb5kdc/kudu_127.0.194.193.keytab
--principal=kudu/127.0.194.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.194.212:35045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790394221564982-779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790394221564982-779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790394221564982-779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:44:18.178697 4965 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:44:18.179113 4965 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:44:18.179296 4965 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:44:18.186586 4965 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:44:18.186786 4965 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:44:18.186995 4965 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.194.193
I20260926 03:44:18.196655 4965 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.194.212:35045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790394221564982-779-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790394221564982-779-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790394221564982-779-0/krb5kdc/kudu_127.0.194.193.keytab
--principal=kudu/127.0.194.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.194.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790394221564982-779-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.194.193
--webserver_port=0
--tserver_master_addrs=127.0.194.254:42827
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.194.193
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790394221564982-779-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:44:18.198189 4965 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:44:18.199533 4965 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:44:18.212196 4970 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:44:18.212121 4973 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:44:18.212224 4971 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:44:18.213508 4965 server_base.cc:1034] running on GCE node
I20260926 03:44:18.214080 4965 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:44:18.215066 4965 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:44:18.216302 4965 hybrid_clock.cc:648] HybridClock initialized: now 1790394258216251 us; error 42 us; skew 500 ppm
I20260926 03:44:18.216642 4965 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:44:18 dist-test-slave-n2th krb5kdc[4860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394258, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:44:18.219713 4965 init.cc:377] Logged in from keytab as kudu/127.0.194.193@KRBTEST.COM (short username kudu)
I20260926 03:44:18.220834 4965 fs_manager.cc:359] Metadata directory not provided
I20260926 03:44:18.220919 4965 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:44:18.221163 4965 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:44:18.224000 4965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790394221564982-779-0/minicluster-data/ts-0/data/instance:
uuid: "6798f664fc1b441c81286204d444c63a"
format_stamp: "Formatted at 2026-09-26 03:44:18 on dist-test-slave-n2th"
server_key: "5f44b074ab7c9f1fac82d27d5690e047"
server_key_iv: "da521514bf40e7f70cd6ee60e37a8877"
server_key_version: "encryptionkey@0"
I20260926 03:44:18.224758 4965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790394221564982-779-0/minicluster-data/ts-0/wal/instance:
uuid: "6798f664fc1b441c81286204d444c63a"
format_stamp: "Formatted at 2026-09-26 03:44:18 on dist-test-slave-n2th"
server_key: "5f44b074ab7c9f1fac82d27d5690e047"
server_key_iv: "da521514bf40e7f70cd6ee60e37a8877"
server_key_version: "encryptionkey@0"
I20260926 03:44:18.229593 4965 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 03:44:18.232893 4979 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:44:18.234169 4965 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 03:44:18.234297 4965 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790394221564982-779-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790394221564982-779-0/minicluster-data/ts-0/wal
uuid: "6798f664fc1b441c81286204d444c63a"
format_stamp: "Formatted at 2026-09-26 03:44:18 on dist-test-slave-n2th"
server_key: "5f44b074ab7c9f1fac82d27d5690e047"
server_key_iv: "da521514bf40e7f70cd6ee60e37a8877"
server_key_version: "encryptionkey@0"
I20260926 03:44:18.234474 4965 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790394221564982-779-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790394221564982-779-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790394221564982-779-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:44:18.264321 4965 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:44:18.268370 4965 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:44:18.268625 4965 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:44:18.269532 4965 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:44:18.270913 4965 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:44:18.271066 4965 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:44:18.271217 4965 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:44:18.271334 4965 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:44:18.305182 4965 rpc_server.cc:307] RPC server started. Bound to: 127.0.194.193:35921
I20260926 03:44:18.305236 5091 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.194.193:35921 every 8 connection(s)
I20260926 03:44:18.306097 4965 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790394221564982-779-0/minicluster-data/ts-0/data/info.pb
I20260926 03:44:18.311210 779 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu as pid 4965
I20260926 03:44:18.311434 779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790394221564982-779-0/minicluster-data/ts-0/wal/instance
I20260926 03:44:18.312229 779 external_mini_cluster.cc:1428] Setting key 756e9a5e8156b53586a8f8577cbaca6d
Sep 26 03:44:18 dist-test-slave-n2th krb5kdc[4860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394258, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.193@KRBTEST.COM for kudu/127.0.194.254@KRBTEST.COM
I20260926 03:44:18.327963 5094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:18.308691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.193:51427 (local address 127.0.194.254:42827)
0926 03:44:18.309144 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0926 03:44:18.309148 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:44:18.310180 (+ 1032us) server_negotiation.cc:407] Connection header received
0926 03:44:18.312757 (+ 2577us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:44:18.312761 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:44:18.312845 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:44:18.313005 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:44:18.314800 (+ 1795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:18.315524 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:18.317136 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:18.317544 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:18.317687 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:18.321727 (+ 4040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:44:18.321732 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:44:18.321735 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:44:18.321767 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:44:18.324497 (+ 2730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:18.325519 (+ 1022us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:18.325523 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:18.325525 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:18.325638 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:18.326150 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:18.326153 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:18.326155 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:18.326397 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:44:18.326505 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0926 03:44:18.327527 (+ 1022us) server_negotiation.cc:299] Negotiation successful
0926 03:44:18.327696 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":225,"threads_started":1}
I20260926 03:44:18.329789 5095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:18.309293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:42827 (local address 127.0.194.193:51427)
0926 03:44:18.309988 (+ 695us) negotiation.cc:107] Waiting for socket to connect
0926 03:44:18.310041 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 03:44:18.312405 (+ 2364us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:44:18.313249 (+ 844us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:44:18.313258 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:44:18.314015 (+ 757us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:44:18.314623 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:18.314641 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:18.315701 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:18.315707 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:18.316977 (+ 1270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:18.316989 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:18.317798 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:18.317805 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:18.318056 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:18.318840 (+ 784us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:44:18.318868 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:44:18.321405 (+ 2537us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:44:18.324673 (+ 3268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:44:18.324686 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:44:18.324708 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:44:18.325387 (+ 679us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:44:18.325764 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:44:18.325772 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:44:18.325778 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:44:18.326035 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:44:18.326544 (+ 509us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:44:18.326555 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:44:18.327081 (+ 526us) client_negotiation.cc:769] Sending connection context
0926 03:44:18.327440 (+ 359us) client_negotiation.cc:240] Negotiation successful
0926 03:44:18.328078 (+ 638us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":282,"threads_started":1}
WARNING: no policy specified for kudu/127.0.194.194@KRBTEST.COM; defaulting to no policy
I20260926 03:44:18.331586 5092 heartbeater.cc:347] Connected to a master server at 127.0.194.254:42827
I20260926 03:44:18.331952 5092 heartbeater.cc:464] Registering TS with master...
I20260926 03:44:18.332737 5092 heartbeater.cc:511] Master 127.0.194.254:42827 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.194.194@KRBTEST.COM" created.
I20260926 03:44:18.334995 4904 ts_manager.cc:194] Registered new tserver with Master: 6798f664fc1b441c81286204d444c63a (127.0.194.193:35921)
I20260926 03:44:18.336531 4904 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.194.193@KRBTEST.COM'} at 127.0.194.193:51427
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.194.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790394221564982-779-0/krb5kdc/kudu_127.0.194.194.keytab.
Entry for principal kudu/127.0.194.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790394221564982-779-0/krb5kdc/kudu_127.0.194.194.keytab.
I20260926 03:44:18.359318 779 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
/tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790394221564982-779-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790394221564982-779-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790394221564982-779-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790394221564982-779-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.194.194:0
--local_ip_for_outbound_sockets=127.0.194.194
--webserver_interface=127.0.194.194
--webserver_port=0
--tserver_master_addrs=127.0.194.254:42827
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790394221564982-779-0/krb5kdc/kudu_127.0.194.194.keytab
--principal=kudu/127.0.194.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.194.212:35045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790394221564982-779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790394221564982-779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790394221564982-779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:44:18.687824 5098 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:44:18.688107 5098 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:44:18.688263 5098 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:44:18.698076 5098 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:44:18.698230 5098 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:44:18.698433 5098 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.194.194
I20260926 03:44:18.711710 5098 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.194.212:35045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790394221564982-779-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790394221564982-779-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790394221564982-779-0/krb5kdc/kudu_127.0.194.194.keytab
--principal=kudu/127.0.194.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.194.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790394221564982-779-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.194.194
--webserver_port=0
--tserver_master_addrs=127.0.194.254:42827
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.194.194
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790394221564982-779-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:44:18.713105 5098 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:44:18.714484 5098 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:44:18.728369 5104 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:44:18.728340 5103 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:44:18.728636 5106 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:44:18.729226 5098 server_base.cc:1034] running on GCE node
I20260926 03:44:18.729849 5098 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:44:18.730867 5098 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:44:18.732092 5098 hybrid_clock.cc:648] HybridClock initialized: now 1790394258732023 us; error 60 us; skew 500 ppm
I20260926 03:44:18.732471 5098 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:44:18 dist-test-slave-n2th krb5kdc[4860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394258, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:44:18.735380 5098 init.cc:377] Logged in from keytab as kudu/127.0.194.194@KRBTEST.COM (short username kudu)
I20260926 03:44:18.736620 5098 fs_manager.cc:359] Metadata directory not provided
I20260926 03:44:18.736729 5098 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:44:18.737084 5098 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:44:18.739991 5098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790394221564982-779-0/minicluster-data/ts-1/data/instance:
uuid: "ae5fd1a6aaa04a9f9e60b961295e2862"
format_stamp: "Formatted at 2026-09-26 03:44:18 on dist-test-slave-n2th"
server_key: "c83aa72bb3345b228726bbefdba595a3"
server_key_iv: "4a2e2c5431ed0b1a1f295464ea79c447"
server_key_version: "encryptionkey@0"
I20260926 03:44:18.740780 5098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790394221564982-779-0/minicluster-data/ts-1/wal/instance:
uuid: "ae5fd1a6aaa04a9f9e60b961295e2862"
format_stamp: "Formatted at 2026-09-26 03:44:18 on dist-test-slave-n2th"
server_key: "c83aa72bb3345b228726bbefdba595a3"
server_key_iv: "4a2e2c5431ed0b1a1f295464ea79c447"
server_key_version: "encryptionkey@0"
I20260926 03:44:18.745728 5098 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260926 03:44:18.749154 5112 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:44:18.750517 5098 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260926 03:44:18.750716 5098 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790394221564982-779-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790394221564982-779-0/minicluster-data/ts-1/wal
uuid: "ae5fd1a6aaa04a9f9e60b961295e2862"
format_stamp: "Formatted at 2026-09-26 03:44:18 on dist-test-slave-n2th"
server_key: "c83aa72bb3345b228726bbefdba595a3"
server_key_iv: "4a2e2c5431ed0b1a1f295464ea79c447"
server_key_version: "encryptionkey@0"
I20260926 03:44:18.750979 5098 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790394221564982-779-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790394221564982-779-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790394221564982-779-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:44:18.793972 5098 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:44:18.798357 5098 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:44:18.798625 5098 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:44:18.799659 5098 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:44:18.800998 5098 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:44:18.801083 5098 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:44:18.801227 5098 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:44:18.801306 5098 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:44:18.834230 5098 rpc_server.cc:307] RPC server started. Bound to: 127.0.194.194:42775
I20260926 03:44:18.834285 5224 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.194.194:42775 every 8 connection(s)
I20260926 03:44:18.835182 5098 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790394221564982-779-0/minicluster-data/ts-1/data/info.pb
I20260926 03:44:18.840268 779 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu as pid 5098
I20260926 03:44:18.840466 779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790394221564982-779-0/minicluster-data/ts-1/wal/instance
I20260926 03:44:18.840982 779 external_mini_cluster.cc:1428] Setting key e2108d01991e7108ad0c91c5f18fbf89
Sep 26 03:44:18 dist-test-slave-n2th krb5kdc[4860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394258, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.194@KRBTEST.COM for kudu/127.0.194.254@KRBTEST.COM
I20260926 03:44:18.857039 5227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:18.837707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.194:49437 (local address 127.0.194.254:42827)
0926 03:44:18.838184 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0926 03:44:18.838189 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:44:18.839202 (+ 1013us) server_negotiation.cc:407] Connection header received
0926 03:44:18.841206 (+ 2004us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:44:18.841212 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:44:18.841300 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:44:18.841434 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:44:18.843251 (+ 1817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:18.844412 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:18.846014 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:18.846806 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:18.846971 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:18.850982 (+ 4011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:44:18.850987 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:44:18.850990 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:44:18.851023 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:44:18.853611 (+ 2588us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:18.854416 (+ 805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:18.854422 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:18.854425 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:18.854562 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:18.855092 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:18.855097 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:18.855100 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:18.855423 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:44:18.855575 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0926 03:44:18.856421 (+ 846us) server_negotiation.cc:299] Negotiation successful
0926 03:44:18.856647 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":266,"threads_started":1}
I20260926 03:44:18.858024 5228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:18.838278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:42827 (local address 127.0.194.194:49437)
0926 03:44:18.839039 (+ 761us) negotiation.cc:107] Waiting for socket to connect
0926 03:44:18.839084 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 03:44:18.840871 (+ 1787us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:44:18.841692 (+ 821us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:44:18.841705 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:44:18.842301 (+ 596us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:44:18.843078 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:18.843098 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:18.844569 (+ 1471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:18.844573 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:18.845860 (+ 1287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:18.845872 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:18.847034 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:18.847039 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:18.847236 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:18.847990 (+ 754us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:44:18.848014 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:44:18.850690 (+ 2676us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:44:18.853768 (+ 3078us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:44:18.853778 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:44:18.853798 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:44:18.854287 (+ 489us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:44:18.854795 (+ 508us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:44:18.854800 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:44:18.854803 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:44:18.854977 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:44:18.855561 (+ 584us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:44:18.855569 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:44:18.856051 (+ 482us) client_negotiation.cc:769] Sending connection context
0926 03:44:18.856327 (+ 276us) client_negotiation.cc:240] Negotiation successful
0926 03:44:18.856920 (+ 593us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":327,"threads_started":1}
I20260926 03:44:18.859477 5225 heartbeater.cc:347] Connected to a master server at 127.0.194.254:42827
I20260926 03:44:18.859786 5225 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.194.195@KRBTEST.COM; defaulting to no policy
I20260926 03:44:18.860463 5225 heartbeater.cc:511] Master 127.0.194.254:42827 requested a full tablet report, sending...
I20260926 03:44:18.861825 4904 ts_manager.cc:194] Registered new tserver with Master: ae5fd1a6aaa04a9f9e60b961295e2862 (127.0.194.194:42775)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.194.195@KRBTEST.COM" created.
I20260926 03:44:18.862659 4904 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.194.194@KRBTEST.COM'} at 127.0.194.194:49437
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.194.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790394221564982-779-0/krb5kdc/kudu_127.0.194.195.keytab.
Entry for principal kudu/127.0.194.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790394221564982-779-0/krb5kdc/kudu_127.0.194.195.keytab.
I20260926 03:44:18.885205 779 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
/tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790394221564982-779-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790394221564982-779-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790394221564982-779-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790394221564982-779-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.194.195:0
--local_ip_for_outbound_sockets=127.0.194.195
--webserver_interface=127.0.194.195
--webserver_port=0
--tserver_master_addrs=127.0.194.254:42827
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790394221564982-779-0/krb5kdc/kudu_127.0.194.195.keytab
--principal=kudu/127.0.194.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.194.212:35045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790394221564982-779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790394221564982-779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790394221564982-779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:44:19.198288 5231 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:44:19.198591 5231 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:44:19.198782 5231 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:44:19.205853 5231 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:44:19.205993 5231 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:44:19.206164 5231 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.194.195
I20260926 03:44:19.215590 5231 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.194.212:35045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790394221564982-779-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790394221564982-779-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790394221564982-779-0/krb5kdc/kudu_127.0.194.195.keytab
--principal=kudu/127.0.194.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.194.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790394221564982-779-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.194.195
--webserver_port=0
--tserver_master_addrs=127.0.194.254:42827
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.194.195
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790394221564982-779-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:44:19.216820 5231 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:44:19.218060 5231 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:44:19.230398 5236 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:44:19.230374 5237 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:44:19.230497 5239 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:44:19.231067 5231 server_base.cc:1034] running on GCE node
I20260926 03:44:19.231601 5231 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:44:19.232460 5231 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:44:19.233671 5231 hybrid_clock.cc:648] HybridClock initialized: now 1790394259233632 us; error 48 us; skew 500 ppm
I20260926 03:44:19.233968 5231 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:44:19 dist-test-slave-n2th krb5kdc[4860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394259, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:44:19.236877 5231 init.cc:377] Logged in from keytab as kudu/127.0.194.195@KRBTEST.COM (short username kudu)
I20260926 03:44:19.238018 5231 fs_manager.cc:359] Metadata directory not provided
I20260926 03:44:19.238137 5231 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:44:19.238445 5231 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:44:19.241374 5231 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790394221564982-779-0/minicluster-data/ts-2/data/instance:
uuid: "9f2d1c6541864639baa0b5b4ad620628"
format_stamp: "Formatted at 2026-09-26 03:44:19 on dist-test-slave-n2th"
server_key: "169446e2060771ed11617e8c836a4b69"
server_key_iv: "45a20dd5aa1d64598977659623a5380d"
server_key_version: "encryptionkey@0"
I20260926 03:44:19.242161 5231 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790394221564982-779-0/minicluster-data/ts-2/wal/instance:
uuid: "9f2d1c6541864639baa0b5b4ad620628"
format_stamp: "Formatted at 2026-09-26 03:44:19 on dist-test-slave-n2th"
server_key: "169446e2060771ed11617e8c836a4b69"
server_key_iv: "45a20dd5aa1d64598977659623a5380d"
server_key_version: "encryptionkey@0"
I20260926 03:44:19.247139 5231 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260926 03:44:19.250319 5245 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:44:19.251716 5231 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 03:44:19.251896 5231 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790394221564982-779-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790394221564982-779-0/minicluster-data/ts-2/wal
uuid: "9f2d1c6541864639baa0b5b4ad620628"
format_stamp: "Formatted at 2026-09-26 03:44:19 on dist-test-slave-n2th"
server_key: "169446e2060771ed11617e8c836a4b69"
server_key_iv: "45a20dd5aa1d64598977659623a5380d"
server_key_version: "encryptionkey@0"
I20260926 03:44:19.252115 5231 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790394221564982-779-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790394221564982-779-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790394221564982-779-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:44:19.273275 5231 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:44:19.277237 5231 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:44:19.277514 5231 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:44:19.278443 5231 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:44:19.279775 5231 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:44:19.279896 5231 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:44:19.280025 5231 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:44:19.280114 5231 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:44:19.312417 5231 rpc_server.cc:307] RPC server started. Bound to: 127.0.194.195:44883
I20260926 03:44:19.312477 5357 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.194.195:44883 every 8 connection(s)
I20260926 03:44:19.313261 5231 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790394221564982-779-0/minicluster-data/ts-2/data/info.pb
I20260926 03:44:19.323324 779 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu as pid 5231
I20260926 03:44:19.323513 779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790394221564982-779-0/minicluster-data/ts-2/wal/instance
I20260926 03:44:19.324066 779 external_mini_cluster.cc:1428] Setting key 3cbe6cc82c2d5bc73b4b54a6a9406143
Sep 26 03:44:19 dist-test-slave-n2th krb5kdc[4860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394259, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.195@KRBTEST.COM for kudu/127.0.194.254@KRBTEST.COM
I20260926 03:44:19.331998 5227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:19.315622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.195:55719 (local address 127.0.194.254:42827)
0926 03:44:19.315831 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0926 03:44:19.315837 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:44:19.316925 (+ 1088us) server_negotiation.cc:407] Connection header received
0926 03:44:19.318615 (+ 1690us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:44:19.318619 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:44:19.318684 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:44:19.318826 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:44:19.320221 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:19.320955 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:19.322395 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:19.323028 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:19.323167 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:19.326817 (+ 3650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:44:19.326822 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:44:19.326826 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:44:19.326866 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:44:19.329475 (+ 2609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:19.330125 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:19.330129 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:19.330131 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:19.330231 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:19.330577 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:19.330579 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:19.330581 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:19.330883 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:44:19.330997 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0926 03:44:19.331611 (+ 614us) server_negotiation.cc:299] Negotiation successful
0926 03:44:19.331765 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 03:44:19.332970 5360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:19.316110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:42827 (local address 127.0.194.195:55719)
0926 03:44:19.316774 (+ 664us) negotiation.cc:107] Waiting for socket to connect
0926 03:44:19.316812 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 03:44:19.318372 (+ 1560us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:44:19.318993 (+ 621us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:44:19.319001 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:44:19.319510 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:44:19.320063 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:19.320078 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:19.321081 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:19.321086 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:19.322280 (+ 1194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:19.322291 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:19.323225 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:19.323229 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:19.323450 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:19.324207 (+ 757us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:44:19.324228 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:44:19.326566 (+ 2338us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:44:19.329597 (+ 3031us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:44:19.329603 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:44:19.329618 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:44:19.330028 (+ 410us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:44:19.330322 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:44:19.330325 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:44:19.330328 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:44:19.330493 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:44:19.330999 (+ 506us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:44:19.331005 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:44:19.331316 (+ 311us) client_negotiation.cc:769] Sending connection context
0926 03:44:19.331548 (+ 232us) client_negotiation.cc:240] Negotiation successful
0926 03:44:19.331925 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":265,"threads_started":1}
I20260926 03:44:19.334342 5358 heartbeater.cc:347] Connected to a master server at 127.0.194.254:42827
I20260926 03:44:19.334621 5358 heartbeater.cc:464] Registering TS with master...
I20260926 03:44:19.335263 5358 heartbeater.cc:511] Master 127.0.194.254:42827 requested a full tablet report, sending...
I20260926 03:44:19.336611 4904 ts_manager.cc:194] Registered new tserver with Master: 9f2d1c6541864639baa0b5b4ad620628 (127.0.194.195:44883)
I20260926 03:44:19.337395 4904 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.194.195@KRBTEST.COM'} at 127.0.194.195:55719
I20260926 03:44:19.339218 779 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 03:44:19.339852 5092 heartbeater.cc:503] Master 127.0.194.254:42827 was elected leader, sending a full tablet report...
I20260926 03:44:19.355679 5227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:19.344000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33812 (local address 127.0.194.254:42827)
0926 03:44:19.344161 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0926 03:44:19.344165 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:44:19.344633 (+ 468us) server_negotiation.cc:407] Connection header received
0926 03:44:19.344765 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:44:19.344769 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:44:19.344822 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:44:19.344899 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:44:19.346271 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:19.347035 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:19.348210 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:19.348621 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:19.348709 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:19.350800 (+ 2091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:44:19.350805 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:44:19.350808 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:44:19.350836 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:44:19.353043 (+ 2207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:19.353684 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:19.353688 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:19.353690 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:19.353852 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:19.354155 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:19.354159 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:19.354161 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:19.354482 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:44:19.354855 (+ 373us) server_negotiation.cc:1035] Waiting for connection context
0926 03:44:19.355104 (+ 249us) server_negotiation.cc:299] Negotiation successful
0926 03:44:19.355338 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 03:44:19.359759 4903 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33812:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 03:44:19.366657 4903 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 03:44:19.401408 5373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:19.379817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43680 (local address 127.0.194.194:42775)
0926 03:44:19.390648 (+ 10831us) server_negotiation.cc:206] Beginning negotiation
0926 03:44:19.390662 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 03:44:19.391842 (+ 1180us) server_negotiation.cc:407] Connection header received
0926 03:44:19.391987 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:44:19.392000 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:44:19.392396 (+ 396us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:44:19.392678 (+ 282us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:44:19.394608 (+ 1930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:19.396336 (+ 1728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:19.399173 (+ 2837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:19.400127 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:19.400280 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:19.400671 (+ 391us) server_negotiation.cc:1035] Waiting for connection context
0926 03:44:19.400887 (+ 216us) server_negotiation.cc:299] Negotiation successful
0926 03:44:19.401052 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10571,"thread_start_us":11664,"threads_started":1}
I20260926 03:44:19.401408 5372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:19.379534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.194:42775 (local address 127.0.0.1:43680)
0926 03:44:19.391165 (+ 11631us) negotiation.cc:107] Waiting for socket to connect
0926 03:44:19.391192 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0926 03:44:19.391381 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:44:19.393075 (+ 1694us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:44:19.393087 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:44:19.393143 (+ 56us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:44:19.393710 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:19.393727 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:19.396502 (+ 2775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:19.396507 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:19.399007 (+ 2500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:19.399019 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:19.400366 (+ 1347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:19.400372 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:19.400577 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:19.400644 (+ 67us) client_negotiation.cc:769] Sending connection context
0926 03:44:19.400847 (+ 203us) client_negotiation.cc:240] Negotiation successful
0926 03:44:19.401054 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11484,"thread_start_us":11345,"threads_started":1}
I20260926 03:44:19.403856 5374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:19.381122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.193:35921 (local address 127.0.0.1:40034)
0926 03:44:19.394933 (+ 13811us) negotiation.cc:107] Waiting for socket to connect
0926 03:44:19.394955 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 03:44:19.395095 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:44:19.395717 (+ 622us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:44:19.395722 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:44:19.395748 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:44:19.396518 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:19.396529 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:19.398204 (+ 1675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:19.398212 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:19.400856 (+ 2644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:19.400870 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:19.403017 (+ 2147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:19.403023 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:19.403186 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:19.403228 (+ 42us) client_negotiation.cc:769] Sending connection context
0926 03:44:19.403317 (+ 89us) client_negotiation.cc:240] Negotiation successful
0926 03:44:19.403442 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13641,"mutex_wait_us":80,"thread_start_us":13410,"threads_started":1}
I20260926 03:44:19.404567 5375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:19.381511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40034 (local address 127.0.194.193:35921)
0926 03:44:19.394502 (+ 12991us) server_negotiation.cc:206] Beginning negotiation
0926 03:44:19.394514 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 03:44:19.395177 (+ 663us) server_negotiation.cc:407] Connection header received
0926 03:44:19.395289 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:44:19.395296 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:44:19.395506 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:44:19.395733 (+ 227us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:44:19.396754 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:19.398064 (+ 1310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:19.401038 (+ 2974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:19.402487 (+ 1449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:19.402622 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:19.402873 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0926 03:44:19.404098 (+ 1225us) server_negotiation.cc:299] Negotiation successful
0926 03:44:19.404260 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12752,"thread_start_us":12541,"threads_started":1}
I20260926 03:44:19.404910 5370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:19.379031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.195:44883 (local address 127.0.0.1:40054)
0926 03:44:19.380576 (+ 1545us) negotiation.cc:107] Waiting for socket to connect
0926 03:44:19.380639 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0926 03:44:19.380861 (+ 222us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:44:19.395874 (+ 15013us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:44:19.395880 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:44:19.395911 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:44:19.396279 (+ 368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:19.396291 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:19.398569 (+ 2278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:19.398576 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:19.400791 (+ 2215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:19.400807 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:19.404247 (+ 3440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:19.404252 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:19.404372 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:19.404394 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 03:44:19.404457 (+ 63us) client_negotiation.cc:240] Negotiation successful
0926 03:44:19.404539 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1193,"mutex_wait_us":2,"thread_start_us":1025,"threads_started":1}
I20260926 03:44:19.405825 5371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:19.379092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40054 (local address 127.0.194.195:44883)
0926 03:44:19.395237 (+ 16145us) server_negotiation.cc:206] Beginning negotiation
0926 03:44:19.395246 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:44:19.395282 (+ 36us) server_negotiation.cc:407] Connection header received
0926 03:44:19.395386 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:44:19.395393 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:44:19.395627 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:44:19.396519 (+ 892us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:44:19.396837 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:19.398423 (+ 1586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:19.400998 (+ 2575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:19.402487 (+ 1489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:19.402620 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:19.405292 (+ 2672us) server_negotiation.cc:1035] Waiting for connection context
0926 03:44:19.405354 (+ 62us) server_negotiation.cc:299] Negotiation successful
0926 03:44:19.405469 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15971,"thread_start_us":15818,"threads_started":1}
I20260926 03:44:19.405912 5160 tablet_service.cc:1467] Processing CreateTablet for tablet 3cfffa3e91204894a4caf87c392a8726 (DEFAULT_TABLE table=test-table [id=f9e7fa02a8d3476f91973cce5838c1a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:44:19.407384 5160 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3cfffa3e91204894a4caf87c392a8726. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:44:19.407974 5027 tablet_service.cc:1467] Processing CreateTablet for tablet 3cfffa3e91204894a4caf87c392a8726 (DEFAULT_TABLE table=test-table [id=f9e7fa02a8d3476f91973cce5838c1a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:44:19.409148 5027 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3cfffa3e91204894a4caf87c392a8726. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:44:19.409261 5293 tablet_service.cc:1467] Processing CreateTablet for tablet 3cfffa3e91204894a4caf87c392a8726 (DEFAULT_TABLE table=test-table [id=f9e7fa02a8d3476f91973cce5838c1a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:44:19.410468 5293 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3cfffa3e91204894a4caf87c392a8726. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:44:19.420246 5376 tablet_bootstrap.cc:492] T 3cfffa3e91204894a4caf87c392a8726 P 6798f664fc1b441c81286204d444c63a: Bootstrap starting.
I20260926 03:44:19.423456 5377 tablet_bootstrap.cc:492] T 3cfffa3e91204894a4caf87c392a8726 P ae5fd1a6aaa04a9f9e60b961295e2862: Bootstrap starting.
I20260926 03:44:19.423488 5376 tablet_bootstrap.cc:654] T 3cfffa3e91204894a4caf87c392a8726 P 6798f664fc1b441c81286204d444c63a: Neither blocks nor log segments found. Creating new log.
I20260926 03:44:19.424907 5376 log.cc:824] T 3cfffa3e91204894a4caf87c392a8726 P 6798f664fc1b441c81286204d444c63a: Log is configured to *not* fsync() on all Append() calls
I20260926 03:44:19.425803 5378 tablet_bootstrap.cc:492] T 3cfffa3e91204894a4caf87c392a8726 P 9f2d1c6541864639baa0b5b4ad620628: Bootstrap starting.
I20260926 03:44:19.427757 5377 tablet_bootstrap.cc:654] T 3cfffa3e91204894a4caf87c392a8726 P ae5fd1a6aaa04a9f9e60b961295e2862: Neither blocks nor log segments found. Creating new log.
I20260926 03:44:19.428980 5378 tablet_bootstrap.cc:654] T 3cfffa3e91204894a4caf87c392a8726 P 9f2d1c6541864639baa0b5b4ad620628: Neither blocks nor log segments found. Creating new log.
I20260926 03:44:19.429039 5377 log.cc:824] T 3cfffa3e91204894a4caf87c392a8726 P ae5fd1a6aaa04a9f9e60b961295e2862: Log is configured to *not* fsync() on all Append() calls
I20260926 03:44:19.429407 5376 tablet_bootstrap.cc:492] T 3cfffa3e91204894a4caf87c392a8726 P 6798f664fc1b441c81286204d444c63a: No bootstrap required, opened a new log
I20260926 03:44:19.429724 5376 ts_tablet_manager.cc:1397] T 3cfffa3e91204894a4caf87c392a8726 P 6798f664fc1b441c81286204d444c63a: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.004s
I20260926 03:44:19.430148 5378 log.cc:824] T 3cfffa3e91204894a4caf87c392a8726 P 9f2d1c6541864639baa0b5b4ad620628: Log is configured to *not* fsync() on all Append() calls
I20260926 03:44:19.432888 5377 tablet_bootstrap.cc:492] T 3cfffa3e91204894a4caf87c392a8726 P ae5fd1a6aaa04a9f9e60b961295e2862: No bootstrap required, opened a new log
I20260926 03:44:19.433157 5377 ts_tablet_manager.cc:1397] T 3cfffa3e91204894a4caf87c392a8726 P ae5fd1a6aaa04a9f9e60b961295e2862: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20260926 03:44:19.433172 5378 tablet_bootstrap.cc:492] T 3cfffa3e91204894a4caf87c392a8726 P 9f2d1c6541864639baa0b5b4ad620628: No bootstrap required, opened a new log
I20260926 03:44:19.433413 5378 ts_tablet_manager.cc:1397] T 3cfffa3e91204894a4caf87c392a8726 P 9f2d1c6541864639baa0b5b4ad620628: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.005s
I20260926 03:44:19.440503 5377 raft_consensus.cc:348] T 3cfffa3e91204894a4caf87c392a8726 P ae5fd1a6aaa04a9f9e60b961295e2862 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f2d1c6541864639baa0b5b4ad620628" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 44883 } } peers { permanent_uuid: "ae5fd1a6aaa04a9f9e60b961295e2862" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 42775 } } peers { permanent_uuid: "6798f664fc1b441c81286204d444c63a" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 35921 } }
I20260926 03:44:19.440547 5376 raft_consensus.cc:348] T 3cfffa3e91204894a4caf87c392a8726 P 6798f664fc1b441c81286204d444c63a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f2d1c6541864639baa0b5b4ad620628" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 44883 } } peers { permanent_uuid: "ae5fd1a6aaa04a9f9e60b961295e2862" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 42775 } } peers { permanent_uuid: "6798f664fc1b441c81286204d444c63a" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 35921 } }
I20260926 03:44:19.440939 5377 raft_consensus.cc:374] T 3cfffa3e91204894a4caf87c392a8726 P ae5fd1a6aaa04a9f9e60b961295e2862 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:44:19.440981 5376 raft_consensus.cc:374] T 3cfffa3e91204894a4caf87c392a8726 P 6798f664fc1b441c81286204d444c63a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:44:19.441027 5377 raft_consensus.cc:729] T 3cfffa3e91204894a4caf87c392a8726 P ae5fd1a6aaa04a9f9e60b961295e2862 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae5fd1a6aaa04a9f9e60b961295e2862, State: Initialized, Role: FOLLOWER
I20260926 03:44:19.441068 5376 raft_consensus.cc:729] T 3cfffa3e91204894a4caf87c392a8726 P 6798f664fc1b441c81286204d444c63a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6798f664fc1b441c81286204d444c63a, State: Initialized, Role: FOLLOWER
I20260926 03:44:19.441700 5376 consensus_queue.cc:260] T 3cfffa3e91204894a4caf87c392a8726 P 6798f664fc1b441c81286204d444c63a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f2d1c6541864639baa0b5b4ad620628" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 44883 } } peers { permanent_uuid: "ae5fd1a6aaa04a9f9e60b961295e2862" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 42775 } } peers { permanent_uuid: "6798f664fc1b441c81286204d444c63a" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 35921 } }
I20260926 03:44:19.441705 5377 consensus_queue.cc:260] T 3cfffa3e91204894a4caf87c392a8726 P ae5fd1a6aaa04a9f9e60b961295e2862 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f2d1c6541864639baa0b5b4ad620628" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 44883 } } peers { permanent_uuid: "ae5fd1a6aaa04a9f9e60b961295e2862" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 42775 } } peers { permanent_uuid: "6798f664fc1b441c81286204d444c63a" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 35921 } }
I20260926 03:44:19.445039 5378 raft_consensus.cc:348] T 3cfffa3e91204894a4caf87c392a8726 P 9f2d1c6541864639baa0b5b4ad620628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f2d1c6541864639baa0b5b4ad620628" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 44883 } } peers { permanent_uuid: "ae5fd1a6aaa04a9f9e60b961295e2862" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 42775 } } peers { permanent_uuid: "6798f664fc1b441c81286204d444c63a" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 35921 } }
I20260926 03:44:19.445611 5378 raft_consensus.cc:374] T 3cfffa3e91204894a4caf87c392a8726 P 9f2d1c6541864639baa0b5b4ad620628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:44:19.445730 5378 raft_consensus.cc:729] T 3cfffa3e91204894a4caf87c392a8726 P 9f2d1c6541864639baa0b5b4ad620628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f2d1c6541864639baa0b5b4ad620628, State: Initialized, Role: FOLLOWER
I20260926 03:44:19.446563 5378 consensus_queue.cc:260] T 3cfffa3e91204894a4caf87c392a8726 P 9f2d1c6541864639baa0b5b4ad620628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f2d1c6541864639baa0b5b4ad620628" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 44883 } } peers { permanent_uuid: "ae5fd1a6aaa04a9f9e60b961295e2862" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 42775 } } peers { permanent_uuid: "6798f664fc1b441c81286204d444c63a" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 35921 } }
I20260926 03:44:19.448961 5225 heartbeater.cc:503] Master 127.0.194.254:42827 was elected leader, sending a full tablet report...
I20260926 03:44:19.449690 5377 ts_tablet_manager.cc:1428] T 3cfffa3e91204894a4caf87c392a8726 P ae5fd1a6aaa04a9f9e60b961295e2862: Time spent starting tablet: real 0.016s user 0.009s sys 0.008s
I20260926 03:44:19.450246 5376 ts_tablet_manager.cc:1428] T 3cfffa3e91204894a4caf87c392a8726 P 6798f664fc1b441c81286204d444c63a: Time spent starting tablet: real 0.020s user 0.018s sys 0.004s
I20260926 03:44:19.451545 5378 ts_tablet_manager.cc:1428] T 3cfffa3e91204894a4caf87c392a8726 P 9f2d1c6541864639baa0b5b4ad620628: Time spent starting tablet: real 0.018s user 0.011s sys 0.006s
I20260926 03:44:19.451923 5358 heartbeater.cc:503] Master 127.0.194.254:42827 was elected leader, sending a full tablet report...
W20260926 03:44:19.558396 5093 tablet.cc:2369] T 3cfffa3e91204894a4caf87c392a8726 P 6798f664fc1b441c81286204d444c63a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 03:44:19.564720 5359 tablet.cc:2369] T 3cfffa3e91204894a4caf87c392a8726 P 9f2d1c6541864639baa0b5b4ad620628: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 03:44:19.576237 5384 raft_consensus.cc:482] T 3cfffa3e91204894a4caf87c392a8726 P 9f2d1c6541864639baa0b5b4ad620628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 03:44:19.576545 5384 raft_consensus.cc:504] T 3cfffa3e91204894a4caf87c392a8726 P 9f2d1c6541864639baa0b5b4ad620628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f2d1c6541864639baa0b5b4ad620628" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 44883 } } peers { permanent_uuid: "ae5fd1a6aaa04a9f9e60b961295e2862" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 42775 } } peers { permanent_uuid: "6798f664fc1b441c81286204d444c63a" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 35921 } }
I20260926 03:44:19.578131 5384 leader_election.cc:288] T 3cfffa3e91204894a4caf87c392a8726 P 9f2d1c6541864639baa0b5b4ad620628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ae5fd1a6aaa04a9f9e60b961295e2862 (127.0.194.194:42775), 6798f664fc1b441c81286204d444c63a (127.0.194.193:35921)
I20260926 03:44:19.583624 5360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:19.578503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.194:42775 (local address 127.0.194.195:35561)
0926 03:44:19.578766 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0926 03:44:19.578790 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 03:44:19.579004 (+ 214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:44:19.579330 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:44:19.579334 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:44:19.579365 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:44:19.579561 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:19.579574 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:19.580513 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:19.580521 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:19.581811 (+ 1290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:19.581830 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:19.582910 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:19.582917 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:19.583099 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:19.583123 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 03:44:19.583223 (+ 100us) client_negotiation.cc:240] Negotiation successful
0926 03:44:19.583352 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61}
I20260926 03:44:19.583622 5373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:19.578610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.195:35561 (local address 127.0.194.194:42775)
0926 03:44:19.578819 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0926 03:44:19.578825 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:44:19.578918 (+ 93us) server_negotiation.cc:407] Connection header received
0926 03:44:19.579146 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:44:19.579155 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:44:19.579215 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:44:19.579306 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:44:19.579693 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:19.580390 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:19.581969 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:19.582719 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:19.582930 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:19.583009 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 03:44:19.583248 (+ 239us) server_negotiation.cc:299] Negotiation successful
0926 03:44:19.583376 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 03:44:19.584534 5375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:19.578689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.195:41835 (local address 127.0.194.193:35921)
0926 03:44:19.578908 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0926 03:44:19.578916 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:44:19.579688 (+ 772us) server_negotiation.cc:407] Connection header received
0926 03:44:19.579870 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:44:19.579873 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:44:19.579930 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:44:19.580028 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:44:19.580534 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:19.581249 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:19.582879 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:19.583609 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:19.583792 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:19.583873 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0926 03:44:19.584252 (+ 379us) server_negotiation.cc:299] Negotiation successful
0926 03:44:19.584354 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260926 03:44:19.584674 5385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:19.578861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.193:35921 (local address 127.0.194.195:41835)
0926 03:44:19.579577 (+ 716us) negotiation.cc:107] Waiting for socket to connect
0926 03:44:19.579592 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 03:44:19.579731 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:44:19.580038 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:44:19.580043 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:44:19.580064 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:44:19.580371 (+ 307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:19.580378 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:19.581400 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:19.581405 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:19.582704 (+ 1299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:19.582713 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:19.583983 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:19.583988 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:19.584107 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:19.584129 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 03:44:19.584201 (+ 72us) client_negotiation.cc:240] Negotiation successful
0926 03:44:19.584344 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":616,"thread_start_us":454,"threads_started":1}
I20260926 03:44:19.584784 5180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cfffa3e91204894a4caf87c392a8726" candidate_uuid: "9f2d1c6541864639baa0b5b4ad620628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae5fd1a6aaa04a9f9e60b961295e2862" is_pre_election: true
I20260926 03:44:19.585402 5180 raft_consensus.cc:2454] T 3cfffa3e91204894a4caf87c392a8726 P ae5fd1a6aaa04a9f9e60b961295e2862 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f2d1c6541864639baa0b5b4ad620628 in term 0.
I20260926 03:44:19.585707 5047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cfffa3e91204894a4caf87c392a8726" candidate_uuid: "9f2d1c6541864639baa0b5b4ad620628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6798f664fc1b441c81286204d444c63a" is_pre_election: true
I20260926 03:44:19.586126 5248 leader_election.cc:302] T 3cfffa3e91204894a4caf87c392a8726 P 9f2d1c6541864639baa0b5b4ad620628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9f2d1c6541864639baa0b5b4ad620628, ae5fd1a6aaa04a9f9e60b961295e2862; no voters:
I20260926 03:44:19.586231 5047 raft_consensus.cc:2454] T 3cfffa3e91204894a4caf87c392a8726 P 6798f664fc1b441c81286204d444c63a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f2d1c6541864639baa0b5b4ad620628 in term 0.
I20260926 03:44:19.586514 5384 raft_consensus.cc:2781] T 3cfffa3e91204894a4caf87c392a8726 P 9f2d1c6541864639baa0b5b4ad620628 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 03:44:19.586664 5384 raft_consensus.cc:482] T 3cfffa3e91204894a4caf87c392a8726 P 9f2d1c6541864639baa0b5b4ad620628 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 03:44:19.586817 5384 raft_consensus.cc:3037] T 3cfffa3e91204894a4caf87c392a8726 P 9f2d1c6541864639baa0b5b4ad620628 [term 0 FOLLOWER]: Advancing to term 1
W20260926 03:44:19.587004 5226 tablet.cc:2369] T 3cfffa3e91204894a4caf87c392a8726 P ae5fd1a6aaa04a9f9e60b961295e2862: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 03:44:19.588907 5384 raft_consensus.cc:504] T 3cfffa3e91204894a4caf87c392a8726 P 9f2d1c6541864639baa0b5b4ad620628 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f2d1c6541864639baa0b5b4ad620628" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 44883 } } peers { permanent_uuid: "ae5fd1a6aaa04a9f9e60b961295e2862" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 42775 } } peers { permanent_uuid: "6798f664fc1b441c81286204d444c63a" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 35921 } }
I20260926 03:44:19.589537 5384 leader_election.cc:288] T 3cfffa3e91204894a4caf87c392a8726 P 9f2d1c6541864639baa0b5b4ad620628 [CANDIDATE]: Term 1 election: Requested vote from peers ae5fd1a6aaa04a9f9e60b961295e2862 (127.0.194.194:42775), 6798f664fc1b441c81286204d444c63a (127.0.194.193:35921)
I20260926 03:44:19.589927 5180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cfffa3e91204894a4caf87c392a8726" candidate_uuid: "9f2d1c6541864639baa0b5b4ad620628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae5fd1a6aaa04a9f9e60b961295e2862"
I20260926 03:44:19.590018 5047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cfffa3e91204894a4caf87c392a8726" candidate_uuid: "9f2d1c6541864639baa0b5b4ad620628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6798f664fc1b441c81286204d444c63a"
I20260926 03:44:19.590116 5180 raft_consensus.cc:3037] T 3cfffa3e91204894a4caf87c392a8726 P ae5fd1a6aaa04a9f9e60b961295e2862 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:44:19.590260 5047 raft_consensus.cc:3037] T 3cfffa3e91204894a4caf87c392a8726 P 6798f664fc1b441c81286204d444c63a [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:44:19.592396 5180 raft_consensus.cc:2454] T 3cfffa3e91204894a4caf87c392a8726 P ae5fd1a6aaa04a9f9e60b961295e2862 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f2d1c6541864639baa0b5b4ad620628 in term 1.
I20260926 03:44:19.592502 5047 raft_consensus.cc:2454] T 3cfffa3e91204894a4caf87c392a8726 P 6798f664fc1b441c81286204d444c63a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f2d1c6541864639baa0b5b4ad620628 in term 1.
I20260926 03:44:19.592826 5248 leader_election.cc:302] T 3cfffa3e91204894a4caf87c392a8726 P 9f2d1c6541864639baa0b5b4ad620628 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9f2d1c6541864639baa0b5b4ad620628, ae5fd1a6aaa04a9f9e60b961295e2862; no voters:
I20260926 03:44:19.593122 5384 raft_consensus.cc:2781] T 3cfffa3e91204894a4caf87c392a8726 P 9f2d1c6541864639baa0b5b4ad620628 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:44:19.593541 5384 raft_consensus.cc:686] T 3cfffa3e91204894a4caf87c392a8726 P 9f2d1c6541864639baa0b5b4ad620628 [term 1 LEADER]: Becoming Leader. State: Replica: 9f2d1c6541864639baa0b5b4ad620628, State: Running, Role: LEADER
I20260926 03:44:19.594000 5384 consensus_queue.cc:237] T 3cfffa3e91204894a4caf87c392a8726 P 9f2d1c6541864639baa0b5b4ad620628 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f2d1c6541864639baa0b5b4ad620628" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 44883 } } peers { permanent_uuid: "ae5fd1a6aaa04a9f9e60b961295e2862" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 42775 } } peers { permanent_uuid: "6798f664fc1b441c81286204d444c63a" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 35921 } }
I20260926 03:44:19.598320 4903 catalog_manager.cc:5495] T 3cfffa3e91204894a4caf87c392a8726 P 9f2d1c6541864639baa0b5b4ad620628 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9f2d1c6541864639baa0b5b4ad620628 (127.0.194.195). New cstate: current_term: 1 leader_uuid: "9f2d1c6541864639baa0b5b4ad620628" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f2d1c6541864639baa0b5b4ad620628" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 44883 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae5fd1a6aaa04a9f9e60b961295e2862" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 42775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6798f664fc1b441c81286204d444c63a" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 35921 } health_report { overall_health: UNKNOWN } } }
I20260926 03:44:19.628312 5371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:19.623103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40064 (local address 127.0.194.195:44883)
0926 03:44:19.623322 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0926 03:44:19.623328 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:44:19.623435 (+ 107us) server_negotiation.cc:407] Connection header received
0926 03:44:19.623607 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:44:19.623611 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:44:19.623696 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:44:19.623829 (+ 133us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:44:19.624115 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:19.624794 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:19.626206 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:19.626569 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:19.626663 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:19.627129 (+ 466us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:44:19.627648 (+ 519us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:44:19.627761 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0926 03:44:19.627945 (+ 184us) server_negotiation.cc:299] Negotiation successful
0926 03:44:19.628010 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 03:44:19.636282 5047 raft_consensus.cc:1261] T 3cfffa3e91204894a4caf87c392a8726 P 6798f664fc1b441c81286204d444c63a [term 1 FOLLOWER]: Refusing update from remote peer 9f2d1c6541864639baa0b5b4ad620628: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:44:19.636345 5180 raft_consensus.cc:1261] T 3cfffa3e91204894a4caf87c392a8726 P ae5fd1a6aaa04a9f9e60b961295e2862 [term 1 FOLLOWER]: Refusing update from remote peer 9f2d1c6541864639baa0b5b4ad620628: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:44:19.637274 5386 consensus_queue.cc:1035] T 3cfffa3e91204894a4caf87c392a8726 P 9f2d1c6541864639baa0b5b4ad620628 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6798f664fc1b441c81286204d444c63a" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 35921 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:44:19.637682 5384 consensus_queue.cc:1035] T 3cfffa3e91204894a4caf87c392a8726 P 9f2d1c6541864639baa0b5b4ad620628 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae5fd1a6aaa04a9f9e60b961295e2862" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 42775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:44:19.667852 4902 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33812:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:44:19.668162 4902 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33812:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:44:19.671761 4902 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0416f884577f4295b3ca5e95f6f101be: Sending DeleteTablet for 3 replicas of tablet 3cfffa3e91204894a4caf87c392a8726
I20260926 03:44:19.672916 5160 tablet_service.cc:1514] Processing DeleteTablet for tablet 3cfffa3e91204894a4caf87c392a8726 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:44:19 UTC) from {username='kudu', principal='kudu/127.0.194.254@KRBTEST.COM'} at 127.0.0.1:43680
I20260926 03:44:19.672993 5027 tablet_service.cc:1514] Processing DeleteTablet for tablet 3cfffa3e91204894a4caf87c392a8726 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:44:19 UTC) from {username='kudu', principal='kudu/127.0.194.254@KRBTEST.COM'} at 127.0.0.1:40034
I20260926 03:44:19.673870 5293 tablet_service.cc:1514] Processing DeleteTablet for tablet 3cfffa3e91204894a4caf87c392a8726 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:44:19 UTC) from {username='kudu', principal='kudu/127.0.194.254@KRBTEST.COM'} at 127.0.0.1:40054
I20260926 03:44:19.675544 779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu pid 4965
I20260926 03:44:19.685336 5399 tablet_replica.cc:330] T 3cfffa3e91204894a4caf87c392a8726 P ae5fd1a6aaa04a9f9e60b961295e2862: stopping tablet replica
I20260926 03:44:19.685901 5399 raft_consensus.cc:2229] T 3cfffa3e91204894a4caf87c392a8726 P ae5fd1a6aaa04a9f9e60b961295e2862 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:44:19.686482 5399 raft_consensus.cc:2258] T 3cfffa3e91204894a4caf87c392a8726 P ae5fd1a6aaa04a9f9e60b961295e2862 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:44:19.688475 5400 tablet_replica.cc:330] T 3cfffa3e91204894a4caf87c392a8726 P 6798f664fc1b441c81286204d444c63a: stopping tablet replica
I20260926 03:44:19.688943 5400 raft_consensus.cc:2229] T 3cfffa3e91204894a4caf87c392a8726 P 6798f664fc1b441c81286204d444c63a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:44:19.688971 5399 ts_tablet_manager.cc:1905] T 3cfffa3e91204894a4caf87c392a8726 P ae5fd1a6aaa04a9f9e60b961295e2862: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:44:19.689427 5400 raft_consensus.cc:2258] T 3cfffa3e91204894a4caf87c392a8726 P 6798f664fc1b441c81286204d444c63a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:44:19.692097 5400 ts_tablet_manager.cc:1905] T 3cfffa3e91204894a4caf87c392a8726 P 6798f664fc1b441c81286204d444c63a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:44:19.701236 5401 tablet_replica.cc:330] T 3cfffa3e91204894a4caf87c392a8726 P 9f2d1c6541864639baa0b5b4ad620628: stopping tablet replica
I20260926 03:44:19.701728 5401 raft_consensus.cc:2229] T 3cfffa3e91204894a4caf87c392a8726 P 9f2d1c6541864639baa0b5b4ad620628 [term 1 LEADER]: Raft consensus shutting down.
I20260926 03:44:19.702493 5401 raft_consensus.cc:2258] T 3cfffa3e91204894a4caf87c392a8726 P 9f2d1c6541864639baa0b5b4ad620628 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:44:19.704804 5401 ts_tablet_manager.cc:1905] T 3cfffa3e91204894a4caf87c392a8726 P 9f2d1c6541864639baa0b5b4ad620628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:44:19.710204 5399 ts_tablet_manager.cc:1918] T 3cfffa3e91204894a4caf87c392a8726 P ae5fd1a6aaa04a9f9e60b961295e2862: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:44:19.712188 5401 ts_tablet_manager.cc:1918] T 3cfffa3e91204894a4caf87c392a8726 P 9f2d1c6541864639baa0b5b4ad620628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:44:19.712369 5401 log.cc:1196] T 3cfffa3e91204894a4caf87c392a8726 P 9f2d1c6541864639baa0b5b4ad620628: Deleting WAL directory at /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790394221564982-779-0/minicluster-data/ts-2/wal/wals/3cfffa3e91204894a4caf87c392a8726
I20260926 03:44:19.712947 5401 ts_tablet_manager.cc:1939] T 3cfffa3e91204894a4caf87c392a8726 P 9f2d1c6541864639baa0b5b4ad620628: Deleting consensus metadata
Sep 26 03:44:19 dist-test-slave-n2th krb5kdc[4860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394257, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.194.193@KRBTEST.COM
I20260926 03:44:19.714886 5399 log.cc:1196] T 3cfffa3e91204894a4caf87c392a8726 P ae5fd1a6aaa04a9f9e60b961295e2862: Deleting WAL directory at /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790394221564982-779-0/minicluster-data/ts-1/wal/wals/3cfffa3e91204894a4caf87c392a8726
I20260926 03:44:19.715014 4889 catalog_manager.cc:4841] TS 9f2d1c6541864639baa0b5b4ad620628 (127.0.194.195:44883): tablet 3cfffa3e91204894a4caf87c392a8726 (table test-table [id=f9e7fa02a8d3476f91973cce5838c1a0]) successfully deleted
I20260926 03:44:19.715622 5399 ts_tablet_manager.cc:1939] T 3cfffa3e91204894a4caf87c392a8726 P ae5fd1a6aaa04a9f9e60b961295e2862: Deleting consensus metadata
I20260926 03:44:19.715962 5400 ts_tablet_manager.cc:1918] T 3cfffa3e91204894a4caf87c392a8726 P 6798f664fc1b441c81286204d444c63a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:44:19.716168 5400 log.cc:1196] T 3cfffa3e91204894a4caf87c392a8726 P 6798f664fc1b441c81286204d444c63a: Deleting WAL directory at /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790394221564982-779-0/minicluster-data/ts-0/wal/wals/3cfffa3e91204894a4caf87c392a8726
I20260926 03:44:19.716747 5400 ts_tablet_manager.cc:1939] T 3cfffa3e91204894a4caf87c392a8726 P 6798f664fc1b441c81286204d444c63a: Deleting consensus metadata
I20260926 03:44:19.718677 4889 catalog_manager.cc:4841] TS 6798f664fc1b441c81286204d444c63a (127.0.194.193:35921): tablet 3cfffa3e91204894a4caf87c392a8726 (table test-table [id=f9e7fa02a8d3476f91973cce5838c1a0]) successfully deleted
I20260926 03:44:19.720966 4890 catalog_manager.cc:4841] TS ae5fd1a6aaa04a9f9e60b961295e2862 (127.0.194.194:42775): tablet 3cfffa3e91204894a4caf87c392a8726 (table test-table [id=f9e7fa02a8d3476f91973cce5838c1a0]) successfully deleted
I20260926 03:44:19.724705 5375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:19.677170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40038 (local address 127.0.194.193:35921)
0926 03:44:19.677394 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0926 03:44:19.677403 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:44:19.690561 (+ 13158us) server_negotiation.cc:407] Connection header received
0926 03:44:19.690803 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:44:19.690811 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:44:19.690902 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:44:19.690992 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:44:19.700640 (+ 9648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:19.701753 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:19.703829 (+ 2076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:19.704462 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:19.704555 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:19.715897 (+ 11342us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:44:19.715916 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:44:19.715935 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:44:19.715999 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:44:19.721302 (+ 5303us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:19.722410 (+ 1108us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:19.722418 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:19.722420 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:19.722563 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:19.723072 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:19.723075 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:19.723077 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:19.723508 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:44:19.723859 (+ 351us) server_negotiation.cc:1035] Waiting for connection context
0926 03:44:19.724137 (+ 278us) server_negotiation.cc:299] Negotiation successful
0926 03:44:19.724355 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 03:44:19.725340 5087 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:44:19.877133 779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu with pid 4965
I20260926 03:44:19.893977 779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu pid 5098
Sep 26 03:44:19 dist-test-slave-n2th krb5kdc[4860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394257, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.194.194@KRBTEST.COM
I20260926 03:44:19.910468 5373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:19.894837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43684 (local address 127.0.194.194:42775)
0926 03:44:19.895037 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0926 03:44:19.895043 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:44:19.895066 (+ 23us) server_negotiation.cc:407] Connection header received
0926 03:44:19.895189 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:44:19.895195 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:44:19.895289 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:44:19.895400 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:44:19.896557 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:19.897700 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:19.898910 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:19.899533 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:19.899663 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:19.903966 (+ 4303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:44:19.903985 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:44:19.904006 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:44:19.904054 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:44:19.907780 (+ 3726us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:19.908433 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:19.908444 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:19.908447 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:19.908607 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:19.908927 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:19.908932 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:19.908936 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:19.909390 (+ 454us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:44:19.909662 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0926 03:44:19.909864 (+ 202us) server_negotiation.cc:299] Negotiation successful
0926 03:44:19.910095 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 03:44:19.911275 5220 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:44:20.001500 779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu with pid 5098
I20260926 03:44:20.018317 779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu pid 5231
Sep 26 03:44:20 dist-test-slave-n2th krb5kdc[4860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394257, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.194.195@KRBTEST.COM
I20260926 03:44:20.036659 5371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:20.019286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40070 (local address 127.0.194.195:44883)
0926 03:44:20.019482 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0926 03:44:20.019488 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:44:20.019533 (+ 45us) server_negotiation.cc:407] Connection header received
0926 03:44:20.019740 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:44:20.019743 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:44:20.019803 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:44:20.019889 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:44:20.021606 (+ 1717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:20.022407 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:20.024396 (+ 1989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:20.024832 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:20.024924 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:20.030434 (+ 5510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:44:20.030449 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:44:20.030464 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:44:20.030513 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:44:20.033654 (+ 3141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:20.034363 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:20.034371 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:20.034374 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:20.034519 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:20.035030 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:20.035036 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:20.035039 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:20.035398 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:44:20.035730 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0926 03:44:20.036071 (+ 341us) server_negotiation.cc:299] Negotiation successful
0926 03:44:20.036300 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260926 03:44:20.037480 5353 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:44:20.156579 779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu with pid 5231
I20260926 03:44:20.175516 779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu pid 4873
I20260926 03:44:20.176543 4934 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:44:20.281255 779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu with pid 4873
2026-09-26T03:44:20Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4309 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-26T03:44:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:44:20Z Disabled control of system clock
I20260926 03:44:20.347476 779 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
/tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790394221564982-779-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790394221564982-779-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790394221564982-779-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790394221564982-779-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.194.254:44475
--webserver_interface=127.0.194.254
--webserver_port=0
--builtin_ntp_servers=127.0.194.212:39271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.194.254:44475
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260926 03:44:20.662108 5410 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:44:20.662408 5410 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:44:20.662494 5410 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:44:20.669736 5410 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:44:20.669854 5410 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:44:20.669893 5410 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:44:20.669955 5410 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:44:20.670001 5410 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:44:20.679207 5410 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.194.212:39271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790394221564982-779-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790394221564982-779-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.194.254:44475
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.194.254:44475
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790394221564982-779-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.194.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790394221564982-779-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:44:20.680464 5410 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:44:20.682143 5410 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:44:20.693966 5415 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:44:20.693955 5418 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:44:20.693886 5416 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:44:20.694692 5410 server_base.cc:1034] running on GCE node
I20260926 03:44:20.695489 5410 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:44:20.696911 5410 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:44:20.698141 5410 hybrid_clock.cc:648] HybridClock initialized: now 1790394260698087 us; error 38 us; skew 500 ppm
I20260926 03:44:20.698482 5410 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 03:44:20.700747 5410 webserver.cc:460] Webserver started at http://127.0.194.254:43627/ using document root <none> and password file <none>
I20260926 03:44:20.701426 5410 fs_manager.cc:359] Metadata directory not provided
I20260926 03:44:20.701522 5410 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:44:20.701860 5410 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:44:20.704802 5410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790394221564982-779-0/minicluster-data/master-0/data/instance:
uuid: "1d34f2b0de45404d99b918ca09c00100"
format_stamp: "Formatted at 2026-09-26 03:44:20 on dist-test-slave-n2th"
server_key: "15ebdee8fa66e7840e3a8c3bcd295ca7"
server_key_iv: "9d6989016a8ed82bff92d9d3eb9345f9"
server_key_version: "encryptionkey@0"
I20260926 03:44:20.705596 5410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790394221564982-779-0/minicluster-data/master-0/wal/instance:
uuid: "1d34f2b0de45404d99b918ca09c00100"
format_stamp: "Formatted at 2026-09-26 03:44:20 on dist-test-slave-n2th"
server_key: "15ebdee8fa66e7840e3a8c3bcd295ca7"
server_key_iv: "9d6989016a8ed82bff92d9d3eb9345f9"
server_key_version: "encryptionkey@0"
I20260926 03:44:20.710574 5410 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260926 03:44:20.714007 5424 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:44:20.715426 5410 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 03:44:20.715616 5410 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790394221564982-779-0/minicluster-data/master-0/data,/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790394221564982-779-0/minicluster-data/master-0/wal
uuid: "1d34f2b0de45404d99b918ca09c00100"
format_stamp: "Formatted at 2026-09-26 03:44:20 on dist-test-slave-n2th"
server_key: "15ebdee8fa66e7840e3a8c3bcd295ca7"
server_key_iv: "9d6989016a8ed82bff92d9d3eb9345f9"
server_key_version: "encryptionkey@0"
I20260926 03:44:20.715839 5410 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790394221564982-779-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790394221564982-779-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790394221564982-779-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:44:20.721446 5410 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:44:20.722250 5410 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:44:20.722543 5410 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:44:20.740166 5410 rpc_server.cc:307] RPC server started. Bound to: 127.0.194.254:44475
I20260926 03:44:20.740198 5475 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.194.254:44475 every 8 connection(s)
I20260926 03:44:20.741489 5410 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790394221564982-779-0/minicluster-data/master-0/data/info.pb
I20260926 03:44:20.745110 779 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu as pid 5410
I20260926 03:44:20.745313 779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790394221564982-779-0/minicluster-data/master-0/wal/instance
I20260926 03:44:20.745550 5476 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:44:20.746124 779 external_mini_cluster.cc:1428] Setting key 3fc1f4c2d04ccdae2410a611e703768d
I20260926 03:44:20.754370 5479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:20.748228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36516 (local address 127.0.194.254:44475)
0926 03:44:20.749217 (+ 989us) server_negotiation.cc:206] Beginning negotiation
0926 03:44:20.749228 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 03:44:20.749267 (+ 39us) server_negotiation.cc:407] Connection header received
0926 03:44:20.750785 (+ 1518us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:44:20.750808 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:44:20.751240 (+ 432us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:44:20.751641 (+ 401us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:44:20.752737 (+ 1096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:44:20.752743 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:44:20.752756 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:44:20.752778 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:44:20.752838 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:44:20.752853 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:44:20.753097 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0926 03:44:20.753241 (+ 144us) server_negotiation.cc:299] Negotiation successful
0926 03:44:20.753481 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":549,"thread_start_us":344,"threads_started":1}
I20260926 03:44:20.758780 5476 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d34f2b0de45404d99b918ca09c00100: Bootstrap starting.
I20260926 03:44:20.761864 5476 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1d34f2b0de45404d99b918ca09c00100: Neither blocks nor log segments found. Creating new log.
I20260926 03:44:20.763160 5476 log.cc:824] T 00000000000000000000000000000000 P 1d34f2b0de45404d99b918ca09c00100: Log is configured to *not* fsync() on all Append() calls
I20260926 03:44:20.766176 5476 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d34f2b0de45404d99b918ca09c00100: No bootstrap required, opened a new log
I20260926 03:44:20.773749 5476 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1d34f2b0de45404d99b918ca09c00100 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d34f2b0de45404d99b918ca09c00100" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 44475 } }
I20260926 03:44:20.774142 5476 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1d34f2b0de45404d99b918ca09c00100 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:44:20.774250 5476 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1d34f2b0de45404d99b918ca09c00100 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d34f2b0de45404d99b918ca09c00100, State: Initialized, Role: FOLLOWER
I20260926 03:44:20.774876 5476 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1d34f2b0de45404d99b918ca09c00100 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d34f2b0de45404d99b918ca09c00100" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 44475 } }
I20260926 03:44:20.775118 5476 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1d34f2b0de45404d99b918ca09c00100 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:44:20.775252 5476 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1d34f2b0de45404d99b918ca09c00100 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:44:20.775413 5476 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1d34f2b0de45404d99b918ca09c00100 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:44:20.777525 5476 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1d34f2b0de45404d99b918ca09c00100 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d34f2b0de45404d99b918ca09c00100" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 44475 } }
I20260926 03:44:20.778051 5476 leader_election.cc:302] T 00000000000000000000000000000000 P 1d34f2b0de45404d99b918ca09c00100 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1d34f2b0de45404d99b918ca09c00100; no voters:
I20260926 03:44:20.778657 5476 leader_election.cc:288] T 00000000000000000000000000000000 P 1d34f2b0de45404d99b918ca09c00100 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:44:20.778817 5481 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1d34f2b0de45404d99b918ca09c00100 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:44:20.780251 5481 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1d34f2b0de45404d99b918ca09c00100 [term 1 LEADER]: Becoming Leader. State: Replica: 1d34f2b0de45404d99b918ca09c00100, State: Running, Role: LEADER
I20260926 03:44:20.780818 5481 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1d34f2b0de45404d99b918ca09c00100 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d34f2b0de45404d99b918ca09c00100" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 44475 } }
I20260926 03:44:20.781001 5476 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1d34f2b0de45404d99b918ca09c00100 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:44:20.783566 5482 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d34f2b0de45404d99b918ca09c00100 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1d34f2b0de45404d99b918ca09c00100" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d34f2b0de45404d99b918ca09c00100" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 44475 } } }
I20260926 03:44:20.783967 5482 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d34f2b0de45404d99b918ca09c00100 [sys.catalog]: This master's current role is: LEADER
I20260926 03:44:20.783864 5483 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d34f2b0de45404d99b918ca09c00100 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1d34f2b0de45404d99b918ca09c00100. Latest consensus state: current_term: 1 leader_uuid: "1d34f2b0de45404d99b918ca09c00100" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d34f2b0de45404d99b918ca09c00100" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 44475 } } }
I20260926 03:44:20.784168 5483 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d34f2b0de45404d99b918ca09c00100 [sys.catalog]: This master's current role is: LEADER
I20260926 03:44:20.785466 5490 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:44:20.789880 5490 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:44:20.797389 5490 catalog_manager.cc:1347] Generated new cluster ID: 789c79e2e67f4b938f02a79f6a661764
I20260926 03:44:20.797510 5490 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:44:20.824237 5490 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:44:20.824667 5490 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:44:20.830175 5490 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1d34f2b0de45404d99b918ca09c00100: Generated new TSK 0
I20260926 03:44:20.831063 5490 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 03:44:20.840288 779 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
/tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790394221564982-779-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790394221564982-779-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790394221564982-779-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790394221564982-779-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.194.193:0
--local_ip_for_outbound_sockets=127.0.194.193
--webserver_interface=127.0.194.193
--webserver_port=0
--tserver_master_addrs=127.0.194.254:44475
--builtin_ntp_servers=127.0.194.212:39271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260926 03:44:21.162009 5500 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:44:21.162294 5500 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:44:21.162467 5500 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:44:21.169760 5500 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:44:21.169864 5500 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:44:21.170047 5500 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.194.193
I20260926 03:44:21.179451 5500 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.194.212:39271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790394221564982-779-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790394221564982-779-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.194.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790394221564982-779-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.194.193
--webserver_port=0
--tserver_master_addrs=127.0.194.254:44475
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.194.193
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790394221564982-779-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:44:21.180776 5500 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:44:21.182713 5500 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:44:21.195760 5505 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:44:21.195796 5508 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:44:21.195752 5506 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:44:21.196527 5500 server_base.cc:1034] running on GCE node
I20260926 03:44:21.197175 5500 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:44:21.198180 5500 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:44:21.199400 5500 hybrid_clock.cc:648] HybridClock initialized: now 1790394261199354 us; error 37 us; skew 500 ppm
I20260926 03:44:21.199712 5500 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 03:44:21.201897 5500 webserver.cc:460] Webserver started at http://127.0.194.193:46157/ using document root <none> and password file <none>
I20260926 03:44:21.202584 5500 fs_manager.cc:359] Metadata directory not provided
I20260926 03:44:21.202706 5500 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:44:21.203073 5500 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:44:21.205935 5500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790394221564982-779-0/minicluster-data/ts-0/data/instance:
uuid: "c225bd6a18fa46c08dc1433022681e04"
format_stamp: "Formatted at 2026-09-26 03:44:21 on dist-test-slave-n2th"
server_key: "442226e9e0866ed9c4525447b4fb8d34"
server_key_iv: "76de41d8d60bf635d69b5b827dd8409a"
server_key_version: "encryptionkey@0"
I20260926 03:44:21.206791 5500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790394221564982-779-0/minicluster-data/ts-0/wal/instance:
uuid: "c225bd6a18fa46c08dc1433022681e04"
format_stamp: "Formatted at 2026-09-26 03:44:21 on dist-test-slave-n2th"
server_key: "442226e9e0866ed9c4525447b4fb8d34"
server_key_iv: "76de41d8d60bf635d69b5b827dd8409a"
server_key_version: "encryptionkey@0"
I20260926 03:44:21.212051 5500 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260926 03:44:21.215512 5514 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:44:21.216862 5500 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 03:44:21.217080 5500 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790394221564982-779-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790394221564982-779-0/minicluster-data/ts-0/wal
uuid: "c225bd6a18fa46c08dc1433022681e04"
format_stamp: "Formatted at 2026-09-26 03:44:21 on dist-test-slave-n2th"
server_key: "442226e9e0866ed9c4525447b4fb8d34"
server_key_iv: "76de41d8d60bf635d69b5b827dd8409a"
server_key_version: "encryptionkey@0"
I20260926 03:44:21.217316 5500 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790394221564982-779-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790394221564982-779-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790394221564982-779-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:44:21.223064 5500 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:44:21.223841 5500 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:44:21.224129 5500 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:44:21.225028 5500 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:44:21.226383 5500 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:44:21.226497 5500 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:44:21.226626 5500 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:44:21.226711 5500 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:44:21.258474 5500 rpc_server.cc:307] RPC server started. Bound to: 127.0.194.193:43345
I20260926 03:44:21.258524 5626 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.194.193:43345 every 8 connection(s)
I20260926 03:44:21.259804 5500 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790394221564982-779-0/minicluster-data/ts-0/data/info.pb
I20260926 03:44:21.267704 5629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:21.262335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.193:38247 (local address 127.0.194.254:44475)
0926 03:44:21.262807 (+ 472us) server_negotiation.cc:206] Beginning negotiation
0926 03:44:21.262813 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:44:21.264030 (+ 1217us) server_negotiation.cc:407] Connection header received
0926 03:44:21.265655 (+ 1625us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:44:21.265660 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:44:21.265712 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:44:21.265811 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:44:21.267121 (+ 1310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:44:21.267125 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:44:21.267128 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:44:21.267143 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:44:21.267180 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:44:21.267192 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:44:21.267288 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0926 03:44:21.267444 (+ 156us) server_negotiation.cc:299] Negotiation successful
0926 03:44:21.267493 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":214,"threads_started":1}
I20260926 03:44:21.268656 5630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:21.262980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:44475 (local address 127.0.194.193:38247)
0926 03:44:21.263853 (+ 873us) negotiation.cc:107] Waiting for socket to connect
0926 03:44:21.263909 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0926 03:44:21.265392 (+ 1483us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:44:21.266005 (+ 613us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:44:21.266012 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:44:21.266103 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:44:21.266821 (+ 718us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 03:44:21.266823 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:44:21.266861 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 03:44:21.266963 (+ 102us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:44:21.267275 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:44:21.267282 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:44:21.267287 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 03:44:21.267507 (+ 220us) client_negotiation.cc:240] Negotiation successful
0926 03:44:21.267714 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":252,"threads_started":1}
I20260926 03:44:21.269606 779 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu as pid 5500
I20260926 03:44:21.269806 779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790394221564982-779-0/minicluster-data/ts-0/wal/instance
I20260926 03:44:21.270112 5627 heartbeater.cc:347] Connected to a master server at 127.0.194.254:44475
I20260926 03:44:21.270409 5627 heartbeater.cc:464] Registering TS with master...
I20260926 03:44:21.270498 779 external_mini_cluster.cc:1428] Setting key 6e080cc3caac44f3ee787e6d9ed1a71e
I20260926 03:44:21.271090 5627 heartbeater.cc:511] Master 127.0.194.254:44475 requested a full tablet report, sending...
I20260926 03:44:21.273136 5441 ts_manager.cc:194] Registered new tserver with Master: c225bd6a18fa46c08dc1433022681e04 (127.0.194.193:43345)
I20260926 03:44:21.273166 779 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
/tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790394221564982-779-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790394221564982-779-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790394221564982-779-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790394221564982-779-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.194.194:0
--local_ip_for_outbound_sockets=127.0.194.194
--webserver_interface=127.0.194.194
--webserver_port=0
--tserver_master_addrs=127.0.194.254:44475
--builtin_ntp_servers=127.0.194.212:39271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260926 03:44:21.590969 5631 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:44:21.591252 5631 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:44:21.591344 5631 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:44:21.598685 5631 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:44:21.598820 5631 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:44:21.598932 5631 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.194.194
I20260926 03:44:21.608268 5631 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.194.212:39271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790394221564982-779-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790394221564982-779-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.194.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790394221564982-779-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.194.194
--webserver_port=0
--tserver_master_addrs=127.0.194.254:44475
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.194.194
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790394221564982-779-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:44:21.609498 5631 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:44:21.611276 5631 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:44:21.624313 5636 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:44:21.624356 5637 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:44:21.624328 5639 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:44:21.625401 5631 server_base.cc:1034] running on GCE node
I20260926 03:44:21.625964 5631 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:44:21.626899 5631 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:44:21.628091 5631 hybrid_clock.cc:648] HybridClock initialized: now 1790394261628045 us; error 45 us; skew 500 ppm
I20260926 03:44:21.628489 5631 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 03:44:21.630684 5631 webserver.cc:460] Webserver started at http://127.0.194.194:46879/ using document root <none> and password file <none>
I20260926 03:44:21.631394 5631 fs_manager.cc:359] Metadata directory not provided
I20260926 03:44:21.631538 5631 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:44:21.631845 5631 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:44:21.634670 5631 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790394221564982-779-0/minicluster-data/ts-1/data/instance:
uuid: "b6fe308bb1b347b085b69e720a82d66f"
format_stamp: "Formatted at 2026-09-26 03:44:21 on dist-test-slave-n2th"
server_key: "8070ff2f0c2322db86e1596211f7f1d4"
server_key_iv: "452f4478a00ca6a47ed131adb43a4b74"
server_key_version: "encryptionkey@0"
I20260926 03:44:21.635500 5631 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790394221564982-779-0/minicluster-data/ts-1/wal/instance:
uuid: "b6fe308bb1b347b085b69e720a82d66f"
format_stamp: "Formatted at 2026-09-26 03:44:21 on dist-test-slave-n2th"
server_key: "8070ff2f0c2322db86e1596211f7f1d4"
server_key_iv: "452f4478a00ca6a47ed131adb43a4b74"
server_key_version: "encryptionkey@0"
I20260926 03:44:21.640524 5631 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 03:44:21.644426 5645 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:44:21.645744 5631 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 03:44:21.645921 5631 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790394221564982-779-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790394221564982-779-0/minicluster-data/ts-1/wal
uuid: "b6fe308bb1b347b085b69e720a82d66f"
format_stamp: "Formatted at 2026-09-26 03:44:21 on dist-test-slave-n2th"
server_key: "8070ff2f0c2322db86e1596211f7f1d4"
server_key_iv: "452f4478a00ca6a47ed131adb43a4b74"
server_key_version: "encryptionkey@0"
I20260926 03:44:21.646157 5631 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790394221564982-779-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790394221564982-779-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790394221564982-779-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:44:21.651629 5631 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:44:21.652406 5631 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:44:21.652664 5631 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:44:21.653537 5631 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:44:21.654940 5631 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:44:21.655054 5631 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:44:21.655177 5631 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:44:21.655260 5631 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:44:21.687182 5631 rpc_server.cc:307] RPC server started. Bound to: 127.0.194.194:42143
I20260926 03:44:21.687235 5757 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.194.194:42143 every 8 connection(s)
I20260926 03:44:21.688552 5631 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790394221564982-779-0/minicluster-data/ts-1/data/info.pb
I20260926 03:44:21.692142 779 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu as pid 5631
I20260926 03:44:21.692324 779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790394221564982-779-0/minicluster-data/ts-1/wal/instance
I20260926 03:44:21.692757 779 external_mini_cluster.cc:1428] Setting key aa5ad505260908f1accb73483bdddbfe
I20260926 03:44:21.695580 779 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
/tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790394221564982-779-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790394221564982-779-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790394221564982-779-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790394221564982-779-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.194.195:0
--local_ip_for_outbound_sockets=127.0.194.195
--webserver_interface=127.0.194.195
--webserver_port=0
--tserver_master_addrs=127.0.194.254:44475
--builtin_ntp_servers=127.0.194.212:39271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260926 03:44:21.697377 5629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:21.691188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.194:46029 (local address 127.0.194.254:44475)
0926 03:44:21.691352 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0926 03:44:21.691356 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:44:21.693023 (+ 1667us) server_negotiation.cc:407] Connection header received
0926 03:44:21.694980 (+ 1957us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:44:21.694986 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:44:21.695047 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:44:21.695157 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:44:21.696532 (+ 1375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:44:21.696541 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:44:21.696549 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:44:21.696575 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:44:21.696619 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:44:21.696639 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:44:21.696763 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0926 03:44:21.696939 (+ 176us) server_negotiation.cc:299] Negotiation successful
0926 03:44:21.697008 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 03:44:21.698288 5760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:21.691760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:44475 (local address 127.0.194.194:46029)
0926 03:44:21.692843 (+ 1083us) negotiation.cc:107] Waiting for socket to connect
0926 03:44:21.692894 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 03:44:21.694669 (+ 1775us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:44:21.695359 (+ 690us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:44:21.695368 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:44:21.695465 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:44:21.696226 (+ 761us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 03:44:21.696228 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:44:21.696269 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 03:44:21.696360 (+ 91us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:44:21.696744 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:44:21.696752 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:44:21.696757 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 03:44:21.696973 (+ 216us) client_negotiation.cc:240] Negotiation successful
0926 03:44:21.697260 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":540,"thread_start_us":353,"threads_started":1}
I20260926 03:44:21.699775 5758 heartbeater.cc:347] Connected to a master server at 127.0.194.254:44475
I20260926 03:44:21.700081 5758 heartbeater.cc:464] Registering TS with master...
I20260926 03:44:21.700680 5758 heartbeater.cc:511] Master 127.0.194.254:44475 requested a full tablet report, sending...
I20260926 03:44:21.702134 5441 ts_manager.cc:194] Registered new tserver with Master: b6fe308bb1b347b085b69e720a82d66f (127.0.194.194:42143)
W20260926 03:44:22.048032 5761 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:44:22.048305 5761 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:44:22.048460 5761 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:44:22.055800 5761 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:44:22.055913 5761 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:44:22.056087 5761 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.194.195
I20260926 03:44:22.065198 5761 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.194.212:39271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790394221564982-779-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790394221564982-779-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.194.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790394221564982-779-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.194.195
--webserver_port=0
--tserver_master_addrs=127.0.194.254:44475
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.194.195
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790394221564982-779-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:44:22.066494 5761 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:44:22.068320 5761 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:44:22.081362 5766 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:44:22.081280 5769 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:44:22.081336 5767 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:44:22.082376 5761 server_base.cc:1034] running on GCE node
I20260926 03:44:22.083107 5761 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:44:22.084072 5761 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:44:22.085319 5761 hybrid_clock.cc:648] HybridClock initialized: now 1790394262085254 us; error 49 us; skew 500 ppm
I20260926 03:44:22.085682 5761 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 03:44:22.087981 5761 webserver.cc:460] Webserver started at http://127.0.194.195:45279/ using document root <none> and password file <none>
I20260926 03:44:22.088694 5761 fs_manager.cc:359] Metadata directory not provided
I20260926 03:44:22.088822 5761 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:44:22.089133 5761 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:44:22.092064 5761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790394221564982-779-0/minicluster-data/ts-2/data/instance:
uuid: "d00cf862bdb94fa193b8128ecbcdfd38"
format_stamp: "Formatted at 2026-09-26 03:44:22 on dist-test-slave-n2th"
server_key: "290bfb20a1a541a8d2e266b33aa2a2cd"
server_key_iv: "87bafc54b696107481e0103a05f09d57"
server_key_version: "encryptionkey@0"
I20260926 03:44:22.092852 5761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790394221564982-779-0/minicluster-data/ts-2/wal/instance:
uuid: "d00cf862bdb94fa193b8128ecbcdfd38"
format_stamp: "Formatted at 2026-09-26 03:44:22 on dist-test-slave-n2th"
server_key: "290bfb20a1a541a8d2e266b33aa2a2cd"
server_key_iv: "87bafc54b696107481e0103a05f09d57"
server_key_version: "encryptionkey@0"
I20260926 03:44:22.097893 5761 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 03:44:22.101264 5775 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:44:22.102587 5761 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 03:44:22.102823 5761 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790394221564982-779-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790394221564982-779-0/minicluster-data/ts-2/wal
uuid: "d00cf862bdb94fa193b8128ecbcdfd38"
format_stamp: "Formatted at 2026-09-26 03:44:22 on dist-test-slave-n2th"
server_key: "290bfb20a1a541a8d2e266b33aa2a2cd"
server_key_iv: "87bafc54b696107481e0103a05f09d57"
server_key_version: "encryptionkey@0"
I20260926 03:44:22.103051 5761 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790394221564982-779-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790394221564982-779-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790394221564982-779-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:44:22.108808 5761 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:44:22.109603 5761 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:44:22.109876 5761 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:44:22.110850 5761 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:44:22.112267 5761 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:44:22.112387 5761 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:44:22.112520 5761 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:44:22.112615 5761 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:44:22.144600 5761 rpc_server.cc:307] RPC server started. Bound to: 127.0.194.195:38735
I20260926 03:44:22.144646 5887 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.194.195:38735 every 8 connection(s)
I20260926 03:44:22.145937 5761 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790394221564982-779-0/minicluster-data/ts-2/data/info.pb
I20260926 03:44:22.146852 779 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu as pid 5761
I20260926 03:44:22.147027 779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790394221564982-779-0/minicluster-data/ts-2/wal/instance
I20260926 03:44:22.147601 779 external_mini_cluster.cc:1428] Setting key 0321d10a8b8f6b82f8c84c99108888e7
I20260926 03:44:22.154793 5629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:22.148934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.195:41191 (local address 127.0.194.254:44475)
0926 03:44:22.149132 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0926 03:44:22.149140 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:44:22.151119 (+ 1979us) server_negotiation.cc:407] Connection header received
0926 03:44:22.152753 (+ 1634us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:44:22.152757 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:44:22.152806 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:44:22.152897 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:44:22.154186 (+ 1289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:44:22.154190 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:44:22.154193 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:44:22.154208 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:44:22.154236 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:44:22.154247 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:44:22.154343 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0926 03:44:22.154474 (+ 131us) server_negotiation.cc:299] Negotiation successful
0926 03:44:22.154526 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 03:44:22.155752 5890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:22.149718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:44475 (local address 127.0.194.195:41191)
0926 03:44:22.150976 (+ 1258us) negotiation.cc:107] Waiting for socket to connect
0926 03:44:22.151029 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 03:44:22.152474 (+ 1445us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:44:22.153096 (+ 622us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:44:22.153104 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:44:22.153197 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:44:22.153910 (+ 713us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 03:44:22.153913 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:44:22.153952 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 03:44:22.154036 (+ 84us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:44:22.154313 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:44:22.154317 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:44:22.154322 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 03:44:22.154519 (+ 197us) client_negotiation.cc:240] Negotiation successful
0926 03:44:22.154720 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":678,"thread_start_us":333,"threads_started":1}
I20260926 03:44:22.157019 5888 heartbeater.cc:347] Connected to a master server at 127.0.194.254:44475
I20260926 03:44:22.157289 5888 heartbeater.cc:464] Registering TS with master...
I20260926 03:44:22.157830 5888 heartbeater.cc:511] Master 127.0.194.254:44475 requested a full tablet report, sending...
I20260926 03:44:22.159169 5441 ts_manager.cc:194] Registered new tserver with Master: d00cf862bdb94fa193b8128ecbcdfd38 (127.0.194.195:38735)
I20260926 03:44:22.162659 779 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 03:44:22.169453 5629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:22.166967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36526 (local address 127.0.194.254:44475)
0926 03:44:22.167169 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0926 03:44:22.167176 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:44:22.167655 (+ 479us) server_negotiation.cc:407] Connection header received
0926 03:44:22.167795 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:44:22.167799 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:44:22.167848 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:44:22.167927 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:44:22.168827 (+ 900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:44:22.168830 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:44:22.168833 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:44:22.168845 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:44:22.168871 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:44:22.168881 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:44:22.168969 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0926 03:44:22.169113 (+ 144us) server_negotiation.cc:299] Negotiation successful
0926 03:44:22.169204 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 03:44:22.173416 5441 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36526:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 03:44:22.180229 5441 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 03:44:22.196892 5902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:22.192083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43856 (local address 127.0.194.193:43345)
0926 03:44:22.193822 (+ 1739us) server_negotiation.cc:206] Beginning negotiation
0926 03:44:22.193832 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 03:44:22.193864 (+ 32us) server_negotiation.cc:407] Connection header received
0926 03:44:22.193938 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:44:22.193945 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:44:22.194193 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:44:22.194354 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:44:22.195748 (+ 1394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:44:22.195761 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:44:22.195780 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:44:22.195813 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:44:22.195879 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:44:22.195905 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:44:22.196017 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0926 03:44:22.196344 (+ 327us) server_negotiation.cc:299] Negotiation successful
0926 03:44:22.196563 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1567,"thread_start_us":1444,"threads_started":1}
I20260926 03:44:22.197495 5901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:22.191830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.193:43345 (local address 127.0.0.1:43856)
0926 03:44:22.193078 (+ 1248us) negotiation.cc:107] Waiting for socket to connect
0926 03:44:22.193126 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 03:44:22.193297 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:44:22.194402 (+ 1105us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:44:22.194413 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:44:22.194532 (+ 119us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:44:22.195383 (+ 851us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 03:44:22.195388 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:44:22.195432 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 03:44:22.195529 (+ 97us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:44:22.196113 (+ 584us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:44:22.196125 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:44:22.196132 (+ 7us) client_negotiation.cc:769] Sending connection context
0926 03:44:22.196343 (+ 211us) client_negotiation.cc:240] Negotiation successful
0926 03:44:22.196492 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":918,"threads_started":1}
I20260926 03:44:22.199218 5903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:22.192432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51752 (local address 127.0.194.195:38735)
0926 03:44:22.196836 (+ 4404us) server_negotiation.cc:206] Beginning negotiation
0926 03:44:22.196846 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 03:44:22.196884 (+ 38us) server_negotiation.cc:407] Connection header received
0926 03:44:22.196951 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:44:22.196961 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:44:22.197188 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:44:22.197346 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:44:22.198401 (+ 1055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:44:22.198407 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:44:22.198417 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:44:22.198429 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:44:22.198468 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:44:22.198481 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:44:22.198659 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0926 03:44:22.198722 (+ 63us) server_negotiation.cc:299] Negotiation successful
0926 03:44:22.198844 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4241,"thread_start_us":4072,"threads_started":1}
I20260926 03:44:22.199245 5904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:22.192116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.195:38735 (local address 127.0.0.1:51752)
0926 03:44:22.196157 (+ 4041us) negotiation.cc:107] Waiting for socket to connect
0926 03:44:22.196172 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 03:44:22.196276 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:44:22.197345 (+ 1069us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:44:22.197350 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:44:22.197372 (+ 22us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:44:22.198113 (+ 741us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 03:44:22.198116 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:44:22.198133 (+ 17us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 03:44:22.198179 (+ 46us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:44:22.198584 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:44:22.198588 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:44:22.198593 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 03:44:22.198723 (+ 130us) client_negotiation.cc:240] Negotiation successful
0926 03:44:22.198797 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3901,"thread_start_us":392,"threads_started":1}
I20260926 03:44:22.201387 5905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:22.194563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45744 (local address 127.0.194.194:42143)
0926 03:44:22.195725 (+ 1162us) server_negotiation.cc:206] Beginning negotiation
0926 03:44:22.195737 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 03:44:22.197976 (+ 2239us) server_negotiation.cc:407] Connection header received
0926 03:44:22.198256 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:44:22.198269 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:44:22.198534 (+ 265us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:44:22.198685 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:44:22.200595 (+ 1910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:44:22.200603 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:44:22.200616 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:44:22.200645 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:44:22.200709 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:44:22.200733 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:44:22.200852 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0926 03:44:22.201077 (+ 225us) server_negotiation.cc:299] Negotiation successful
0926 03:44:22.201170 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":980,"thread_start_us":852,"threads_started":1}
I20260926 03:44:22.201880 5901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:22.194119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.194:42143 (local address 127.0.0.1:45744)
0926 03:44:22.197875 (+ 3756us) negotiation.cc:107] Waiting for socket to connect
0926 03:44:22.197894 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 03:44:22.198069 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:44:22.199805 (+ 1736us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:44:22.199809 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:44:22.199831 (+ 22us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:44:22.200400 (+ 569us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 03:44:22.200403 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:44:22.200418 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 03:44:22.200464 (+ 46us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:44:22.200902 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:44:22.200909 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:44:22.200916 (+ 7us) client_negotiation.cc:769] Sending connection context
0926 03:44:22.201655 (+ 739us) client_negotiation.cc:240] Negotiation successful
0926 03:44:22.201704 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3655,"thread_start_us":1778,"threads_started":1}
I20260926 03:44:22.203655 5823 tablet_service.cc:1467] Processing CreateTablet for tablet 475ee78e1e7847b9b395be22924e84dd (DEFAULT_TABLE table=test-table [id=501633b288bf4c38bd5abd7680902773]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:44:22.204028 5562 tablet_service.cc:1467] Processing CreateTablet for tablet 475ee78e1e7847b9b395be22924e84dd (DEFAULT_TABLE table=test-table [id=501633b288bf4c38bd5abd7680902773]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:44:22.205236 5562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 475ee78e1e7847b9b395be22924e84dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:44:22.205236 5823 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 475ee78e1e7847b9b395be22924e84dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:44:22.204952 5693 tablet_service.cc:1467] Processing CreateTablet for tablet 475ee78e1e7847b9b395be22924e84dd (DEFAULT_TABLE table=test-table [id=501633b288bf4c38bd5abd7680902773]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:44:22.206066 5693 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 475ee78e1e7847b9b395be22924e84dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:44:22.217631 5906 tablet_bootstrap.cc:492] T 475ee78e1e7847b9b395be22924e84dd P b6fe308bb1b347b085b69e720a82d66f: Bootstrap starting.
I20260926 03:44:22.221370 5906 tablet_bootstrap.cc:654] T 475ee78e1e7847b9b395be22924e84dd P b6fe308bb1b347b085b69e720a82d66f: Neither blocks nor log segments found. Creating new log.
I20260926 03:44:22.223039 5906 log.cc:824] T 475ee78e1e7847b9b395be22924e84dd P b6fe308bb1b347b085b69e720a82d66f: Log is configured to *not* fsync() on all Append() calls
I20260926 03:44:22.225533 5908 tablet_bootstrap.cc:492] T 475ee78e1e7847b9b395be22924e84dd P d00cf862bdb94fa193b8128ecbcdfd38: Bootstrap starting.
I20260926 03:44:22.225975 5906 tablet_bootstrap.cc:492] T 475ee78e1e7847b9b395be22924e84dd P b6fe308bb1b347b085b69e720a82d66f: No bootstrap required, opened a new log
I20260926 03:44:22.226235 5906 ts_tablet_manager.cc:1397] T 475ee78e1e7847b9b395be22924e84dd P b6fe308bb1b347b085b69e720a82d66f: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260926 03:44:22.228468 5908 tablet_bootstrap.cc:654] T 475ee78e1e7847b9b395be22924e84dd P d00cf862bdb94fa193b8128ecbcdfd38: Neither blocks nor log segments found. Creating new log.
I20260926 03:44:22.229568 5908 log.cc:824] T 475ee78e1e7847b9b395be22924e84dd P d00cf862bdb94fa193b8128ecbcdfd38: Log is configured to *not* fsync() on all Append() calls
I20260926 03:44:22.230924 5907 tablet_bootstrap.cc:492] T 475ee78e1e7847b9b395be22924e84dd P c225bd6a18fa46c08dc1433022681e04: Bootstrap starting.
I20260926 03:44:22.234421 5907 tablet_bootstrap.cc:654] T 475ee78e1e7847b9b395be22924e84dd P c225bd6a18fa46c08dc1433022681e04: Neither blocks nor log segments found. Creating new log.
I20260926 03:44:22.235265 5906 raft_consensus.cc:348] T 475ee78e1e7847b9b395be22924e84dd P b6fe308bb1b347b085b69e720a82d66f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c225bd6a18fa46c08dc1433022681e04" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 43345 } } peers { permanent_uuid: "d00cf862bdb94fa193b8128ecbcdfd38" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 38735 } } peers { permanent_uuid: "b6fe308bb1b347b085b69e720a82d66f" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 42143 } }
I20260926 03:44:22.235631 5907 log.cc:824] T 475ee78e1e7847b9b395be22924e84dd P c225bd6a18fa46c08dc1433022681e04: Log is configured to *not* fsync() on all Append() calls
I20260926 03:44:22.235750 5906 raft_consensus.cc:374] T 475ee78e1e7847b9b395be22924e84dd P b6fe308bb1b347b085b69e720a82d66f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:44:22.235845 5906 raft_consensus.cc:729] T 475ee78e1e7847b9b395be22924e84dd P b6fe308bb1b347b085b69e720a82d66f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6fe308bb1b347b085b69e720a82d66f, State: Initialized, Role: FOLLOWER
I20260926 03:44:22.236538 5906 consensus_queue.cc:260] T 475ee78e1e7847b9b395be22924e84dd P b6fe308bb1b347b085b69e720a82d66f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c225bd6a18fa46c08dc1433022681e04" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 43345 } } peers { permanent_uuid: "d00cf862bdb94fa193b8128ecbcdfd38" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 38735 } } peers { permanent_uuid: "b6fe308bb1b347b085b69e720a82d66f" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 42143 } }
I20260926 03:44:22.237301 5908 tablet_bootstrap.cc:492] T 475ee78e1e7847b9b395be22924e84dd P d00cf862bdb94fa193b8128ecbcdfd38: No bootstrap required, opened a new log
I20260926 03:44:22.237578 5908 ts_tablet_manager.cc:1397] T 475ee78e1e7847b9b395be22924e84dd P d00cf862bdb94fa193b8128ecbcdfd38: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.007s
I20260926 03:44:22.237624 5758 heartbeater.cc:503] Master 127.0.194.254:44475 was elected leader, sending a full tablet report...
I20260926 03:44:22.238227 5906 ts_tablet_manager.cc:1428] T 475ee78e1e7847b9b395be22924e84dd P b6fe308bb1b347b085b69e720a82d66f: Time spent starting tablet: real 0.012s user 0.013s sys 0.000s
I20260926 03:44:22.238669 5907 tablet_bootstrap.cc:492] T 475ee78e1e7847b9b395be22924e84dd P c225bd6a18fa46c08dc1433022681e04: No bootstrap required, opened a new log
I20260926 03:44:22.238956 5907 ts_tablet_manager.cc:1397] T 475ee78e1e7847b9b395be22924e84dd P c225bd6a18fa46c08dc1433022681e04: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.000s
I20260926 03:44:22.248499 5908 raft_consensus.cc:348] T 475ee78e1e7847b9b395be22924e84dd P d00cf862bdb94fa193b8128ecbcdfd38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c225bd6a18fa46c08dc1433022681e04" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 43345 } } peers { permanent_uuid: "d00cf862bdb94fa193b8128ecbcdfd38" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 38735 } } peers { permanent_uuid: "b6fe308bb1b347b085b69e720a82d66f" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 42143 } }
I20260926 03:44:22.249009 5908 raft_consensus.cc:374] T 475ee78e1e7847b9b395be22924e84dd P d00cf862bdb94fa193b8128ecbcdfd38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:44:22.249137 5908 raft_consensus.cc:729] T 475ee78e1e7847b9b395be22924e84dd P d00cf862bdb94fa193b8128ecbcdfd38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d00cf862bdb94fa193b8128ecbcdfd38, State: Initialized, Role: FOLLOWER
I20260926 03:44:22.249498 5907 raft_consensus.cc:348] T 475ee78e1e7847b9b395be22924e84dd P c225bd6a18fa46c08dc1433022681e04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c225bd6a18fa46c08dc1433022681e04" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 43345 } } peers { permanent_uuid: "d00cf862bdb94fa193b8128ecbcdfd38" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 38735 } } peers { permanent_uuid: "b6fe308bb1b347b085b69e720a82d66f" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 42143 } }
I20260926 03:44:22.249946 5907 raft_consensus.cc:374] T 475ee78e1e7847b9b395be22924e84dd P c225bd6a18fa46c08dc1433022681e04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:44:22.249827 5908 consensus_queue.cc:260] T 475ee78e1e7847b9b395be22924e84dd P d00cf862bdb94fa193b8128ecbcdfd38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c225bd6a18fa46c08dc1433022681e04" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 43345 } } peers { permanent_uuid: "d00cf862bdb94fa193b8128ecbcdfd38" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 38735 } } peers { permanent_uuid: "b6fe308bb1b347b085b69e720a82d66f" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 42143 } }
I20260926 03:44:22.250041 5907 raft_consensus.cc:729] T 475ee78e1e7847b9b395be22924e84dd P c225bd6a18fa46c08dc1433022681e04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c225bd6a18fa46c08dc1433022681e04, State: Initialized, Role: FOLLOWER
I20260926 03:44:22.251258 5888 heartbeater.cc:503] Master 127.0.194.254:44475 was elected leader, sending a full tablet report...
I20260926 03:44:22.252000 5908 ts_tablet_manager.cc:1428] T 475ee78e1e7847b9b395be22924e84dd P d00cf862bdb94fa193b8128ecbcdfd38: Time spent starting tablet: real 0.014s user 0.014s sys 0.002s
I20260926 03:44:22.250900 5907 consensus_queue.cc:260] T 475ee78e1e7847b9b395be22924e84dd P c225bd6a18fa46c08dc1433022681e04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c225bd6a18fa46c08dc1433022681e04" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 43345 } } peers { permanent_uuid: "d00cf862bdb94fa193b8128ecbcdfd38" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 38735 } } peers { permanent_uuid: "b6fe308bb1b347b085b69e720a82d66f" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 42143 } }
I20260926 03:44:22.253484 5907 ts_tablet_manager.cc:1428] T 475ee78e1e7847b9b395be22924e84dd P c225bd6a18fa46c08dc1433022681e04: Time spent starting tablet: real 0.014s user 0.008s sys 0.004s
I20260926 03:44:22.253851 5627 heartbeater.cc:503] Master 127.0.194.254:44475 was elected leader, sending a full tablet report...
W20260926 03:44:22.261894 5628 tablet.cc:2369] T 475ee78e1e7847b9b395be22924e84dd P c225bd6a18fa46c08dc1433022681e04: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 03:44:22.298141 5913 raft_consensus.cc:482] T 475ee78e1e7847b9b395be22924e84dd P d00cf862bdb94fa193b8128ecbcdfd38 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 03:44:22.298456 5913 raft_consensus.cc:504] T 475ee78e1e7847b9b395be22924e84dd P d00cf862bdb94fa193b8128ecbcdfd38 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c225bd6a18fa46c08dc1433022681e04" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 43345 } } peers { permanent_uuid: "d00cf862bdb94fa193b8128ecbcdfd38" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 38735 } } peers { permanent_uuid: "b6fe308bb1b347b085b69e720a82d66f" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 42143 } }
I20260926 03:44:22.300545 5913 leader_election.cc:288] T 475ee78e1e7847b9b395be22924e84dd P d00cf862bdb94fa193b8128ecbcdfd38 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c225bd6a18fa46c08dc1433022681e04 (127.0.194.193:43345), b6fe308bb1b347b085b69e720a82d66f (127.0.194.194:42143)
I20260926 03:44:22.303248 5902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:22.300844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.195:33165 (local address 127.0.194.193:43345)
0926 03:44:22.301039 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0926 03:44:22.301045 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:44:22.301304 (+ 259us) server_negotiation.cc:407] Connection header received
0926 03:44:22.301478 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:44:22.301483 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:44:22.301538 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:44:22.301638 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:44:22.302637 (+ 999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:44:22.302642 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:44:22.302647 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:44:22.302665 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:44:22.302705 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:44:22.302724 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:44:22.302862 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0926 03:44:22.302947 (+ 85us) server_negotiation.cc:299] Negotiation successful
0926 03:44:22.303005 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 03:44:22.303344 5890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:22.300921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.193:43345 (local address 127.0.194.195:33165)
0926 03:44:22.301209 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0926 03:44:22.301224 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 03:44:22.301347 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:44:22.301664 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:44:22.301668 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:44:22.301696 (+ 28us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:44:22.302403 (+ 707us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 03:44:22.302405 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:44:22.302420 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 03:44:22.302476 (+ 56us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:44:22.302847 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:44:22.302851 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:44:22.302854 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 03:44:22.302967 (+ 113us) client_negotiation.cc:240] Negotiation successful
0926 03:44:22.303019 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179}
I20260926 03:44:22.303854 5915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:22.300996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.194:42143 (local address 127.0.194.195:58989)
0926 03:44:22.301705 (+ 709us) negotiation.cc:107] Waiting for socket to connect
0926 03:44:22.301719 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 03:44:22.301814 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:44:22.302121 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:44:22.302126 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:44:22.302147 (+ 21us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:44:22.302811 (+ 664us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 03:44:22.302817 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:44:22.302833 (+ 16us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 03:44:22.302873 (+ 40us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:44:22.303236 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:44:22.303242 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:44:22.303249 (+ 7us) client_negotiation.cc:769] Sending connection context
0926 03:44:22.303378 (+ 129us) client_negotiation.cc:240] Negotiation successful
0926 03:44:22.303429 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":578,"thread_start_us":422,"threads_started":1}
I20260926 03:44:22.303987 5905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:22.300965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.195:58989 (local address 127.0.194.194:42143)
0926 03:44:22.301160 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0926 03:44:22.301166 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:44:22.301791 (+ 625us) server_negotiation.cc:407] Connection header received
0926 03:44:22.301926 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:44:22.301933 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:44:22.301994 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:44:22.302102 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:44:22.303014 (+ 912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:44:22.303020 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:44:22.303024 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:44:22.303041 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:44:22.303074 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:44:22.303089 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:44:22.303249 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0926 03:44:22.303644 (+ 395us) server_negotiation.cc:299] Negotiation successful
0926 03:44:22.303711 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260926 03:44:22.304390 5582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "475ee78e1e7847b9b395be22924e84dd" candidate_uuid: "d00cf862bdb94fa193b8128ecbcdfd38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c225bd6a18fa46c08dc1433022681e04" is_pre_election: true
I20260926 03:44:22.304984 5582 raft_consensus.cc:2454] T 475ee78e1e7847b9b395be22924e84dd P c225bd6a18fa46c08dc1433022681e04 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d00cf862bdb94fa193b8128ecbcdfd38 in term 0.
I20260926 03:44:22.304946 5713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "475ee78e1e7847b9b395be22924e84dd" candidate_uuid: "d00cf862bdb94fa193b8128ecbcdfd38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6fe308bb1b347b085b69e720a82d66f" is_pre_election: true
I20260926 03:44:22.305464 5713 raft_consensus.cc:2454] T 475ee78e1e7847b9b395be22924e84dd P b6fe308bb1b347b085b69e720a82d66f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d00cf862bdb94fa193b8128ecbcdfd38 in term 0.
I20260926 03:44:22.305730 5779 leader_election.cc:302] T 475ee78e1e7847b9b395be22924e84dd P d00cf862bdb94fa193b8128ecbcdfd38 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c225bd6a18fa46c08dc1433022681e04, d00cf862bdb94fa193b8128ecbcdfd38; no voters:
I20260926 03:44:22.306186 5913 raft_consensus.cc:2781] T 475ee78e1e7847b9b395be22924e84dd P d00cf862bdb94fa193b8128ecbcdfd38 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 03:44:22.306327 5913 raft_consensus.cc:482] T 475ee78e1e7847b9b395be22924e84dd P d00cf862bdb94fa193b8128ecbcdfd38 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 03:44:22.306457 5913 raft_consensus.cc:3037] T 475ee78e1e7847b9b395be22924e84dd P d00cf862bdb94fa193b8128ecbcdfd38 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:44:22.308724 5913 raft_consensus.cc:504] T 475ee78e1e7847b9b395be22924e84dd P d00cf862bdb94fa193b8128ecbcdfd38 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c225bd6a18fa46c08dc1433022681e04" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 43345 } } peers { permanent_uuid: "d00cf862bdb94fa193b8128ecbcdfd38" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 38735 } } peers { permanent_uuid: "b6fe308bb1b347b085b69e720a82d66f" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 42143 } }
I20260926 03:44:22.309234 5913 leader_election.cc:288] T 475ee78e1e7847b9b395be22924e84dd P d00cf862bdb94fa193b8128ecbcdfd38 [CANDIDATE]: Term 1 election: Requested vote from peers c225bd6a18fa46c08dc1433022681e04 (127.0.194.193:43345), b6fe308bb1b347b085b69e720a82d66f (127.0.194.194:42143)
I20260926 03:44:22.309695 5582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "475ee78e1e7847b9b395be22924e84dd" candidate_uuid: "d00cf862bdb94fa193b8128ecbcdfd38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c225bd6a18fa46c08dc1433022681e04"
I20260926 03:44:22.309695 5713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "475ee78e1e7847b9b395be22924e84dd" candidate_uuid: "d00cf862bdb94fa193b8128ecbcdfd38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6fe308bb1b347b085b69e720a82d66f"
I20260926 03:44:22.309895 5713 raft_consensus.cc:3037] T 475ee78e1e7847b9b395be22924e84dd P b6fe308bb1b347b085b69e720a82d66f [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:44:22.309897 5582 raft_consensus.cc:3037] T 475ee78e1e7847b9b395be22924e84dd P c225bd6a18fa46c08dc1433022681e04 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:44:22.313134 5713 raft_consensus.cc:2454] T 475ee78e1e7847b9b395be22924e84dd P b6fe308bb1b347b085b69e720a82d66f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d00cf862bdb94fa193b8128ecbcdfd38 in term 1.
I20260926 03:44:22.313134 5582 raft_consensus.cc:2454] T 475ee78e1e7847b9b395be22924e84dd P c225bd6a18fa46c08dc1433022681e04 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d00cf862bdb94fa193b8128ecbcdfd38 in term 1.
I20260926 03:44:22.313570 5778 leader_election.cc:302] T 475ee78e1e7847b9b395be22924e84dd P d00cf862bdb94fa193b8128ecbcdfd38 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: b6fe308bb1b347b085b69e720a82d66f, c225bd6a18fa46c08dc1433022681e04, d00cf862bdb94fa193b8128ecbcdfd38; no voters:
I20260926 03:44:22.313926 5913 raft_consensus.cc:2781] T 475ee78e1e7847b9b395be22924e84dd P d00cf862bdb94fa193b8128ecbcdfd38 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:44:22.314371 5913 raft_consensus.cc:686] T 475ee78e1e7847b9b395be22924e84dd P d00cf862bdb94fa193b8128ecbcdfd38 [term 1 LEADER]: Becoming Leader. State: Replica: d00cf862bdb94fa193b8128ecbcdfd38, State: Running, Role: LEADER
I20260926 03:44:22.314901 5913 consensus_queue.cc:237] T 475ee78e1e7847b9b395be22924e84dd P d00cf862bdb94fa193b8128ecbcdfd38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c225bd6a18fa46c08dc1433022681e04" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 43345 } } peers { permanent_uuid: "d00cf862bdb94fa193b8128ecbcdfd38" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 38735 } } peers { permanent_uuid: "b6fe308bb1b347b085b69e720a82d66f" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 42143 } }
I20260926 03:44:22.319416 5441 catalog_manager.cc:5495] T 475ee78e1e7847b9b395be22924e84dd P d00cf862bdb94fa193b8128ecbcdfd38 reported cstate change: term changed from 0 to 1, leader changed from <none> to d00cf862bdb94fa193b8128ecbcdfd38 (127.0.194.195). New cstate: current_term: 1 leader_uuid: "d00cf862bdb94fa193b8128ecbcdfd38" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c225bd6a18fa46c08dc1433022681e04" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 43345 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d00cf862bdb94fa193b8128ecbcdfd38" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 38735 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b6fe308bb1b347b085b69e720a82d66f" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 42143 } health_report { overall_health: UNKNOWN } } }
I20260926 03:44:22.331748 5903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:22.329812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51764 (local address 127.0.194.195:38735)
0926 03:44:22.329963 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0926 03:44:22.329967 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:44:22.330105 (+ 138us) server_negotiation.cc:407] Connection header received
0926 03:44:22.330284 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:44:22.330288 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:44:22.330365 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:44:22.330450 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:44:22.331286 (+ 836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:44:22.331290 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:44:22.331294 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:44:22.331308 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:44:22.331339 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:44:22.331350 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:44:22.331449 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0926 03:44:22.331518 (+ 69us) server_negotiation.cc:299] Negotiation successful
0926 03:44:22.331561 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 03:44:22.339803 5713 raft_consensus.cc:1261] T 475ee78e1e7847b9b395be22924e84dd P b6fe308bb1b347b085b69e720a82d66f [term 1 FOLLOWER]: Refusing update from remote peer d00cf862bdb94fa193b8128ecbcdfd38: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:44:22.339804 5582 raft_consensus.cc:1261] T 475ee78e1e7847b9b395be22924e84dd P c225bd6a18fa46c08dc1433022681e04 [term 1 FOLLOWER]: Refusing update from remote peer d00cf862bdb94fa193b8128ecbcdfd38: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:44:22.340849 5913 consensus_queue.cc:1035] T 475ee78e1e7847b9b395be22924e84dd P d00cf862bdb94fa193b8128ecbcdfd38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6fe308bb1b347b085b69e720a82d66f" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 42143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:44:22.341168 5916 consensus_queue.cc:1035] T 475ee78e1e7847b9b395be22924e84dd P d00cf862bdb94fa193b8128ecbcdfd38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c225bd6a18fa46c08dc1433022681e04" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 43345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:44:22.351512 5919 mvcc.cc:205] Tried to move back new op lower bound from 7333454898534060032 to 7333454898445803520. Current Snapshot: MvccSnapshot[applied={T|T < 7333454898534060032}]
I20260926 03:44:22.355422 5921 mvcc.cc:205] Tried to move back new op lower bound from 7333454898534060032 to 7333454898445803520. Current Snapshot: MvccSnapshot[applied={T|T < 7333454898534060032 or (T in {7333454898534060032})}]
I20260926 03:44:22.368352 5440 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36526:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:44:22.368798 5440 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36526:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:44:22.372948 5440 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1d34f2b0de45404d99b918ca09c00100: Sending DeleteTablet for 3 replicas of tablet 475ee78e1e7847b9b395be22924e84dd
I20260926 03:44:22.374181 5823 tablet_service.cc:1514] Processing DeleteTablet for tablet 475ee78e1e7847b9b395be22924e84dd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:44:22 UTC) from {username='slave'} at 127.0.0.1:51752
I20260926 03:44:22.374207 5693 tablet_service.cc:1514] Processing DeleteTablet for tablet 475ee78e1e7847b9b395be22924e84dd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:44:22 UTC) from {username='slave'} at 127.0.0.1:45744
I20260926 03:44:22.374208 5562 tablet_service.cc:1514] Processing DeleteTablet for tablet 475ee78e1e7847b9b395be22924e84dd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:44:22 UTC) from {username='slave'} at 127.0.0.1:43856
I20260926 03:44:22.375157 5929 tablet_replica.cc:330] T 475ee78e1e7847b9b395be22924e84dd P d00cf862bdb94fa193b8128ecbcdfd38: stopping tablet replica
I20260926 03:44:22.375690 5929 raft_consensus.cc:2229] T 475ee78e1e7847b9b395be22924e84dd P d00cf862bdb94fa193b8128ecbcdfd38 [term 1 LEADER]: Raft consensus shutting down.
I20260926 03:44:22.376556 779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu pid 5500
I20260926 03:44:22.376844 5929 raft_consensus.cc:2258] T 475ee78e1e7847b9b395be22924e84dd P d00cf862bdb94fa193b8128ecbcdfd38 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:44:22.382720 5902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:22.377869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43870 (local address 127.0.194.193:43345)
0926 03:44:22.378068 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0926 03:44:22.378090 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0926 03:44:22.379585 (+ 1495us) server_negotiation.cc:407] Connection header received
0926 03:44:22.379929 (+ 344us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:44:22.379939 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:44:22.380082 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:44:22.380170 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:44:22.381667 (+ 1497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:44:22.381676 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:44:22.381684 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:44:22.381712 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:44:22.381781 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:44:22.381813 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:44:22.381938 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0926 03:44:22.382223 (+ 285us) server_negotiation.cc:299] Negotiation successful
0926 03:44:22.382303 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 03:44:22.382807 5929 ts_tablet_manager.cc:1905] T 475ee78e1e7847b9b395be22924e84dd P d00cf862bdb94fa193b8128ecbcdfd38: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:44:22.383998 5622 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:44:22.392408 5931 tablet_replica.cc:330] T 475ee78e1e7847b9b395be22924e84dd P b6fe308bb1b347b085b69e720a82d66f: stopping tablet replica
I20260926 03:44:22.393018 5931 raft_consensus.cc:2229] T 475ee78e1e7847b9b395be22924e84dd P b6fe308bb1b347b085b69e720a82d66f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:44:22.393687 5931 raft_consensus.cc:2258] T 475ee78e1e7847b9b395be22924e84dd P b6fe308bb1b347b085b69e720a82d66f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:44:22.398377 5929 ts_tablet_manager.cc:1918] T 475ee78e1e7847b9b395be22924e84dd P d00cf862bdb94fa193b8128ecbcdfd38: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:44:22.398593 5929 log.cc:1196] T 475ee78e1e7847b9b395be22924e84dd P d00cf862bdb94fa193b8128ecbcdfd38: Deleting WAL directory at /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790394221564982-779-0/minicluster-data/ts-2/wal/wals/475ee78e1e7847b9b395be22924e84dd
I20260926 03:44:22.399183 5929 ts_tablet_manager.cc:1939] T 475ee78e1e7847b9b395be22924e84dd P d00cf862bdb94fa193b8128ecbcdfd38: Deleting consensus metadata
I20260926 03:44:22.401114 5425 catalog_manager.cc:4841] TS d00cf862bdb94fa193b8128ecbcdfd38 (127.0.194.195:38735): tablet 475ee78e1e7847b9b395be22924e84dd (table test-table [id=501633b288bf4c38bd5abd7680902773]) successfully deleted
I20260926 03:44:22.403280 5931 ts_tablet_manager.cc:1905] T 475ee78e1e7847b9b395be22924e84dd P b6fe308bb1b347b085b69e720a82d66f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:44:22.412022 5931 ts_tablet_manager.cc:1918] T 475ee78e1e7847b9b395be22924e84dd P b6fe308bb1b347b085b69e720a82d66f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:44:22.412221 5931 log.cc:1196] T 475ee78e1e7847b9b395be22924e84dd P b6fe308bb1b347b085b69e720a82d66f: Deleting WAL directory at /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790394221564982-779-0/minicluster-data/ts-1/wal/wals/475ee78e1e7847b9b395be22924e84dd
I20260926 03:44:22.412765 5931 ts_tablet_manager.cc:1939] T 475ee78e1e7847b9b395be22924e84dd P b6fe308bb1b347b085b69e720a82d66f: Deleting consensus metadata
I20260926 03:44:22.414726 5427 catalog_manager.cc:4841] TS b6fe308bb1b347b085b69e720a82d66f (127.0.194.194:42143): tablet 475ee78e1e7847b9b395be22924e84dd (table test-table [id=501633b288bf4c38bd5abd7680902773]) successfully deleted
I20260926 03:44:22.496434 5930 tablet_replica.cc:330] T 475ee78e1e7847b9b395be22924e84dd P c225bd6a18fa46c08dc1433022681e04: stopping tablet replica
I20260926 03:44:22.496753 779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu with pid 5500
W20260926 03:44:22.513880 5428 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.194.193:43345 (error 108)
I20260926 03:44:22.514626 779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu pid 5631
I20260926 03:44:22.516014 5901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:22.515414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.193:43345 (local address 127.0.0.1:43874)
0926 03:44:22.515679 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0926 03:44:22.515818 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.193:43345: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136}
W20260926 03:44:22.516268 5428 catalog_manager.cc:4584] TS c225bd6a18fa46c08dc1433022681e04 (127.0.194.193:43345): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 475ee78e1e7847b9b395be22924e84dd: Network error: Client connection negotiation failed: client connection to 127.0.194.193:43345: connect: Connection refused (error 111)
I20260926 03:44:22.517987 5905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:22.515539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45758 (local address 127.0.194.194:42143)
0926 03:44:22.515713 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0926 03:44:22.515719 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:44:22.515741 (+ 22us) server_negotiation.cc:407] Connection header received
0926 03:44:22.515914 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:44:22.515920 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:44:22.516033 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:44:22.516157 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:44:22.517364 (+ 1207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:44:22.517369 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:44:22.517372 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:44:22.517388 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:44:22.517419 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:44:22.517431 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:44:22.517561 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0926 03:44:22.517656 (+ 95us) server_negotiation.cc:299] Negotiation successful
0926 03:44:22.517717 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 03:44:22.518683 5753 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:44:22.570206 5901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:22.569615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.193:43345 (local address 127.0.0.1:43888)
0926 03:44:22.569914 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0926 03:44:22.570025 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.193:43345: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
I20260926 03:44:22.571388 5901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:22.570978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.193:43345 (local address 127.0.0.1:43896)
0926 03:44:22.571140 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0926 03:44:22.571234 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.193:43345: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260926 03:44:22.606915 779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu with pid 5631
I20260926 03:44:22.629107 779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu pid 5761
I20260926 03:44:22.632241 5903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:22.630048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51768 (local address 127.0.194.195:38735)
0926 03:44:22.630255 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0926 03:44:22.630263 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:44:22.630286 (+ 23us) server_negotiation.cc:407] Connection header received
0926 03:44:22.630379 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:44:22.630394 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:44:22.630442 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:44:22.630555 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:44:22.631536 (+ 981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:44:22.631545 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:44:22.631549 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:44:22.631563 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:44:22.631593 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:44:22.631604 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:44:22.631743 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0926 03:44:22.631832 (+ 89us) server_negotiation.cc:299] Negotiation successful
0926 03:44:22.631893 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 03:44:22.632405 5901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:22.631902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.193:43345 (local address 127.0.0.1:43900)
0926 03:44:22.632161 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0926 03:44:22.632252 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.193:43345: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135}
I20260926 03:44:22.633067 5883 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:44:22.633605 5901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:22.633089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.193:43345 (local address 127.0.0.1:43914)
0926 03:44:22.633327 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0926 03:44:22.633404 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.193:43345: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
I20260926 03:44:22.714066 5901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:22.713527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.193:43345 (local address 127.0.0.1:43918)
0926 03:44:22.713762 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0926 03:44:22.713883 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.193:43345: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260926 03:44:22.715231 5901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:22.714804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.193:43345 (local address 127.0.0.1:43934)
0926 03:44:22.715017 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0926 03:44:22.715102 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.193:43345: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
I20260926 03:44:22.749868 779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu with pid 5761
I20260926 03:44:22.773466 779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu pid 5410
I20260926 03:44:22.774581 5471 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:44:22.880569 779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu with pid 5410
2026-09-26T03:44:22Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2598 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260926 03:44:22.921839 779 mini_oidc.cc:152] Starting JWKS server
I20260926 03:44:22.924904 779 webserver.cc:460] Webserver started at https://127.0.0.1:40607/ using document root <none> and password file <none>
I20260926 03:44:22.925239 779 mini_oidc.cc:189] Starting OIDC Discovery server
I20260926 03:44:22.925890 779 webserver.cc:460] Webserver started at http://127.0.0.1:42571/ using document root <none> and password file <none>
2026-09-26T03:44:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:44:22Z Disabled control of system clock
I20260926 03:44:22.953756 779 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
/tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790394221564982-779-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790394221564982-779-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790394221564982-779-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790394221564982-779-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.194.254:46401
--webserver_interface=127.0.194.254
--webserver_port=0
--builtin_ntp_servers=127.0.194.212:33093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.194.254:46401
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40607/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790394221564982-779-0/testchainca.cert with env {}
W20260926 03:44:23.274704 5943 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:44:23.275089 5943 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:44:23.275264 5943 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:44:23.282539 5943 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:44:23.282698 5943 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:44:23.282809 5943 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:44:23.282919 5943 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:44:23.283000 5943 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 03:44:23.283088 5943 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 03:44:23.283165 5943 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:40607/jwks.json
I20260926 03:44:23.293107 5943 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.194.212:33093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790394221564982-779-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790394221564982-779-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.194.254:46401
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.194.254:46401
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40607/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790394221564982-779-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.194.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790394221564982-779-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790394221564982-779-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:44:23.294466 5943 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:44:23.295879 5943 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:44:23.307417 5949 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:44:23.307493 5951 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:44:23.307533 5948 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:44:23.308622 5943 server_base.cc:1034] running on GCE node
I20260926 03:44:23.309410 5943 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:44:23.310889 5943 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:44:23.312116 5943 hybrid_clock.cc:648] HybridClock initialized: now 1790394263312071 us; error 39 us; skew 500 ppm
I20260926 03:44:23.312472 5943 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 03:44:23.314718 5943 webserver.cc:460] Webserver started at http://127.0.194.254:42619/ using document root <none> and password file <none>
I20260926 03:44:23.315469 5943 fs_manager.cc:359] Metadata directory not provided
I20260926 03:44:23.315609 5943 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:44:23.315905 5943 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:44:23.318864 5943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790394221564982-779-0/minicluster-data/master-0/data/instance:
uuid: "58afd97c97534095b0855757f2ca17b5"
format_stamp: "Formatted at 2026-09-26 03:44:23 on dist-test-slave-n2th"
server_key: "82f2fbe404d11c3ecf441f50cedafc87"
server_key_iv: "7a3590dd28760c85d321c9260e055940"
server_key_version: "encryptionkey@0"
I20260926 03:44:23.319674 5943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790394221564982-779-0/minicluster-data/master-0/wal/instance:
uuid: "58afd97c97534095b0855757f2ca17b5"
format_stamp: "Formatted at 2026-09-26 03:44:23 on dist-test-slave-n2th"
server_key: "82f2fbe404d11c3ecf441f50cedafc87"
server_key_iv: "7a3590dd28760c85d321c9260e055940"
server_key_version: "encryptionkey@0"
I20260926 03:44:23.324868 5943 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.002s
I20260926 03:44:23.328575 5957 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:44:23.329988 5943 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 03:44:23.330191 5943 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790394221564982-779-0/minicluster-data/master-0/data,/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790394221564982-779-0/minicluster-data/master-0/wal
uuid: "58afd97c97534095b0855757f2ca17b5"
format_stamp: "Formatted at 2026-09-26 03:44:23 on dist-test-slave-n2th"
server_key: "82f2fbe404d11c3ecf441f50cedafc87"
server_key_iv: "7a3590dd28760c85d321c9260e055940"
server_key_version: "encryptionkey@0"
I20260926 03:44:23.330422 5943 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790394221564982-779-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790394221564982-779-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790394221564982-779-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:44:23.351157 5943 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:44:23.352034 5943 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:44:23.352348 5943 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:44:23.372669 5943 rpc_server.cc:307] RPC server started. Bound to: 127.0.194.254:46401
I20260926 03:44:23.372695 6008 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.194.254:46401 every 8 connection(s)
I20260926 03:44:23.374015 5943 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790394221564982-779-0/minicluster-data/master-0/data/info.pb
I20260926 03:44:23.377775 6009 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:44:23.382157 779 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu as pid 5943
I20260926 03:44:23.382371 779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790394221564982-779-0/minicluster-data/master-0/wal/instance
I20260926 03:44:23.383211 779 external_mini_cluster.cc:1428] Setting key a8d8d1ce2efb3614e56e357ae4f0d6ad
I20260926 03:44:23.389745 6009 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58afd97c97534095b0855757f2ca17b5: Bootstrap starting.
I20260926 03:44:23.392875 6009 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 58afd97c97534095b0855757f2ca17b5: Neither blocks nor log segments found. Creating new log.
I20260926 03:44:23.393924 6009 log.cc:824] T 00000000000000000000000000000000 P 58afd97c97534095b0855757f2ca17b5: Log is configured to *not* fsync() on all Append() calls
I20260926 03:44:23.397392 6009 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58afd97c97534095b0855757f2ca17b5: No bootstrap required, opened a new log
I20260926 03:44:23.397708 6012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:23.385137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59626 (local address 127.0.194.254:46401)
0926 03:44:23.386087 (+ 950us) server_negotiation.cc:206] Beginning negotiation
0926 03:44:23.386098 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 03:44:23.386140 (+ 42us) server_negotiation.cc:407] Connection header received
0926 03:44:23.388468 (+ 2328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:44:23.388492 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:44:23.388962 (+ 470us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:44:23.389358 (+ 396us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:44:23.389918 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:23.391303 (+ 1385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:23.392646 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:23.393393 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:23.393604 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:23.394953 (+ 1349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:44:23.394960 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:44:23.394974 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:44:23.395631 (+ 657us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:44:23.395731 (+ 100us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:44:23.395756 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:44:23.395888 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0926 03:44:23.396106 (+ 218us) server_negotiation.cc:299] Negotiation successful
0926 03:44:23.396431 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":525,"thread_start_us":303,"threads_started":1}
I20260926 03:44:23.401459 779 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 03:44:23.406358 6009 raft_consensus.cc:348] T 00000000000000000000000000000000 P 58afd97c97534095b0855757f2ca17b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58afd97c97534095b0855757f2ca17b5" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 46401 } }
I20260926 03:44:23.406721 6009 raft_consensus.cc:374] T 00000000000000000000000000000000 P 58afd97c97534095b0855757f2ca17b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:44:23.406910 6009 raft_consensus.cc:729] T 00000000000000000000000000000000 P 58afd97c97534095b0855757f2ca17b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58afd97c97534095b0855757f2ca17b5, State: Initialized, Role: FOLLOWER
I20260926 03:44:23.407506 6009 consensus_queue.cc:260] T 00000000000000000000000000000000 P 58afd97c97534095b0855757f2ca17b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58afd97c97534095b0855757f2ca17b5" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 46401 } }
I20260926 03:44:23.407737 6009 raft_consensus.cc:388] T 00000000000000000000000000000000 P 58afd97c97534095b0855757f2ca17b5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:44:23.407894 6009 raft_consensus.cc:482] T 00000000000000000000000000000000 P 58afd97c97534095b0855757f2ca17b5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:44:23.408094 6009 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 58afd97c97534095b0855757f2ca17b5 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:44:23.410377 6009 raft_consensus.cc:504] T 00000000000000000000000000000000 P 58afd97c97534095b0855757f2ca17b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58afd97c97534095b0855757f2ca17b5" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 46401 } }
I20260926 03:44:23.410995 6009 leader_election.cc:302] T 00000000000000000000000000000000 P 58afd97c97534095b0855757f2ca17b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 58afd97c97534095b0855757f2ca17b5; no voters:
I20260926 03:44:23.411558 6009 leader_election.cc:288] T 00000000000000000000000000000000 P 58afd97c97534095b0855757f2ca17b5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:44:23.411710 6014 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 58afd97c97534095b0855757f2ca17b5 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:44:23.412449 6014 raft_consensus.cc:686] T 00000000000000000000000000000000 P 58afd97c97534095b0855757f2ca17b5 [term 1 LEADER]: Becoming Leader. State: Replica: 58afd97c97534095b0855757f2ca17b5, State: Running, Role: LEADER
I20260926 03:44:23.412930 6014 consensus_queue.cc:237] T 00000000000000000000000000000000 P 58afd97c97534095b0855757f2ca17b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58afd97c97534095b0855757f2ca17b5" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 46401 } }
I20260926 03:44:23.413744 6009 sys_catalog.cc:564] T 00000000000000000000000000000000 P 58afd97c97534095b0855757f2ca17b5 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:44:23.415808 6016 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58afd97c97534095b0855757f2ca17b5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 58afd97c97534095b0855757f2ca17b5. Latest consensus state: current_term: 1 leader_uuid: "58afd97c97534095b0855757f2ca17b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58afd97c97534095b0855757f2ca17b5" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 46401 } } }
I20260926 03:44:23.416314 6016 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58afd97c97534095b0855757f2ca17b5 [sys.catalog]: This master's current role is: LEADER
I20260926 03:44:23.416523 6015 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58afd97c97534095b0855757f2ca17b5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "58afd97c97534095b0855757f2ca17b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58afd97c97534095b0855757f2ca17b5" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 46401 } } }
I20260926 03:44:23.417105 6015 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58afd97c97534095b0855757f2ca17b5 [sys.catalog]: This master's current role is: LEADER
I20260926 03:44:23.418781 6021 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:44:23.424017 6021 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:44:23.433422 6021 catalog_manager.cc:1347] Generated new cluster ID: 430837a6a9514bbea0416c67d0ff3de0
I20260926 03:44:23.433548 6021 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:44:23.448728 6021 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:44:23.450608 6021 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:44:23.461733 6021 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 58afd97c97534095b0855757f2ca17b5: Generated new TSK 0
I20260926 03:44:23.462874 6021 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260926 03:44:23.505507 6039 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260926 03:44:23.505686 6039 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260926 03:44:23.506898 6012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:23.484542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59640 (local address 127.0.194.254:46401)
0926 03:44:23.484741 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0926 03:44:23.484748 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:44:23.485362 (+ 614us) server_negotiation.cc:407] Connection header received
0926 03:44:23.485654 (+ 292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:44:23.485665 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:44:23.485756 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:44:23.485923 (+ 167us) server_negotiation.cc:226] Negotiated authn=JWT
0926 03:44:23.486424 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:23.487639 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:23.489042 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:23.489747 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:23.489873 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:23.490412 (+ 539us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 03:44:23.506172 (+ 15760us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40607/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0926 03:44:23.506555 (+ 383us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:59640: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40607/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":70}
W20260926 03:44:23.507023 6012 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:59640: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40607/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260926 03:44:23.508419 6038 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:44:23.484543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:46401 (local address 127.0.0.1:59640)
0926 03:44:23.485252 (+ 709us) negotiation.cc:107] Waiting for socket to connect
0926 03:44:23.485269 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 03:44:23.485427 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:44:23.485951 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:44:23.485955 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:44:23.485964 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0926 03:44:23.486277 (+ 313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:23.486284 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:23.487769 (+ 1485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:23.487772 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:23.488898 (+ 1126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:23.488909 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:23.489932 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:23.489937 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:23.490054 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:23.490116 (+ 62us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 03:44:23.506501 (+ 16385us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40607/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0926 03:44:23.506848 (+ 347us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.194.254:46401: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40607/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":362,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.0.194.254:46401: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40607/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260926 03:44:23.510478 779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu pid 5943
I20260926 03:44:23.511736 6004 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:44:23.606455 779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu with pid 5943
2026-09-26T03:44:23Z chronyd exiting
I20260926 03:44:23.657023 779 test_util.cc:175] -----------------------------------------------
I20260926 03:44:23.657176 779 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790394221564982-779-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (739 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260926 03:44:23.660841 779 mini_oidc.cc:152] Starting JWKS server
I20260926 03:44:23.662624 779 webserver.cc:460] Webserver started at https://127.0.0.1:40175/ using document root <none> and password file <none>
I20260926 03:44:23.662878 779 mini_oidc.cc:189] Starting OIDC Discovery server
I20260926 03:44:23.663514 779 webserver.cc:460] Webserver started at http://127.0.0.1:44763/ using document root <none> and password file <none>
2026-09-26T03:44:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:44:23Z Disabled control of system clock
I20260926 03:44:23.690721 779 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
/tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790394221564982-779-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790394221564982-779-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790394221564982-779-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790394221564982-779-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.194.254:44605
--webserver_interface=127.0.194.254
--webserver_port=0
--builtin_ntp_servers=127.0.194.212:41435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.194.254:44605
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40175/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790394221564982-779-0/testchainca.cert with env {}
W20260926 03:44:24.019179 6046 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:44:24.019471 6046 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:44:24.019647 6046 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:44:24.026973 6046 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:44:24.027096 6046 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:44:24.027194 6046 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:44:24.027287 6046 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:44:24.027379 6046 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 03:44:24.027490 6046 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 03:44:24.027573 6046 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:40175/jwks.json
I20260926 03:44:24.037883 6046 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.194.212:41435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790394221564982-779-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790394221564982-779-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.194.254:44605
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.194.254:44605
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40175/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790394221564982-779-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.194.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790394221564982-779-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790394221564982-779-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:44:24.039253 6046 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:44:24.040540 6046 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:44:24.052572 6051 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:44:24.052574 6052 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:44:24.052554 6054 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:44:24.056363 6046 server_base.cc:1034] running on GCE node
I20260926 03:44:24.057190 6046 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:44:24.058641 6046 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:44:24.059840 6046 hybrid_clock.cc:648] HybridClock initialized: now 1790394264059781 us; error 64 us; skew 500 ppm
I20260926 03:44:24.060163 6046 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 03:44:24.068022 6046 webserver.cc:460] Webserver started at http://127.0.194.254:37279/ using document root <none> and password file <none>
I20260926 03:44:24.068728 6046 fs_manager.cc:359] Metadata directory not provided
I20260926 03:44:24.068823 6046 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:44:24.069097 6046 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:44:24.072028 6046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790394221564982-779-0/minicluster-data/master-0/data/instance:
uuid: "c497e6ca916e4ab0b8aa706199f93e30"
format_stamp: "Formatted at 2026-09-26 03:44:24 on dist-test-slave-n2th"
server_key: "80edcca5cee9381b519c7ec33d035c24"
server_key_iv: "3aeb21cc00839ee237756ed4b9f84ebb"
server_key_version: "encryptionkey@0"
I20260926 03:44:24.072782 6046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790394221564982-779-0/minicluster-data/master-0/wal/instance:
uuid: "c497e6ca916e4ab0b8aa706199f93e30"
format_stamp: "Formatted at 2026-09-26 03:44:24 on dist-test-slave-n2th"
server_key: "80edcca5cee9381b519c7ec33d035c24"
server_key_iv: "3aeb21cc00839ee237756ed4b9f84ebb"
server_key_version: "encryptionkey@0"
I20260926 03:44:24.077780 6046 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 03:44:24.081101 6060 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:44:24.082389 6046 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260926 03:44:24.082572 6046 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790394221564982-779-0/minicluster-data/master-0/data,/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790394221564982-779-0/minicluster-data/master-0/wal
uuid: "c497e6ca916e4ab0b8aa706199f93e30"
format_stamp: "Formatted at 2026-09-26 03:44:24 on dist-test-slave-n2th"
server_key: "80edcca5cee9381b519c7ec33d035c24"
server_key_iv: "3aeb21cc00839ee237756ed4b9f84ebb"
server_key_version: "encryptionkey@0"
I20260926 03:44:24.082824 6046 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790394221564982-779-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790394221564982-779-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790394221564982-779-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:44:24.094002 6046 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:44:24.094866 6046 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:44:24.095165 6046 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:44:24.113636 6046 rpc_server.cc:307] RPC server started. Bound to: 127.0.194.254:44605
I20260926 03:44:24.113660 6111 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.194.254:44605 every 8 connection(s)
I20260926 03:44:24.115065 6046 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790394221564982-779-0/minicluster-data/master-0/data/info.pb
I20260926 03:44:24.118863 6112 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:44:24.119282 779 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu as pid 6046
I20260926 03:44:24.119477 779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790394221564982-779-0/minicluster-data/master-0/wal/instance
I20260926 03:44:24.120437 779 external_mini_cluster.cc:1428] Setting key aac7e68fe4c312317bb654e91729760e
I20260926 03:44:24.131701 6112 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c497e6ca916e4ab0b8aa706199f93e30: Bootstrap starting.
I20260926 03:44:24.134088 6115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:24.122422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57378 (local address 127.0.194.254:44605)
0926 03:44:24.123509 (+ 1087us) server_negotiation.cc:206] Beginning negotiation
0926 03:44:24.123520 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 03:44:24.123563 (+ 43us) server_negotiation.cc:407] Connection header received
0926 03:44:24.125653 (+ 2090us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:44:24.125677 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:44:24.126184 (+ 507us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:44:24.126631 (+ 447us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:44:24.127227 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:24.128460 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:24.129777 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:24.130437 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:24.130630 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:24.131865 (+ 1235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:44:24.131877 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:44:24.131898 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:44:24.132436 (+ 538us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:44:24.132510 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:44:24.132530 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:44:24.132652 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0926 03:44:24.132848 (+ 196us) server_negotiation.cc:299] Negotiation successful
0926 03:44:24.133157 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":655,"thread_start_us":382,"threads_started":1}
I20260926 03:44:24.134974 6112 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c497e6ca916e4ab0b8aa706199f93e30: Neither blocks nor log segments found. Creating new log.
I20260926 03:44:24.135982 6112 log.cc:824] T 00000000000000000000000000000000 P c497e6ca916e4ab0b8aa706199f93e30: Log is configured to *not* fsync() on all Append() calls
I20260926 03:44:24.137905 779 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 03:44:24.138921 6112 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c497e6ca916e4ab0b8aa706199f93e30: No bootstrap required, opened a new log
I20260926 03:44:24.146827 6112 raft_consensus.cc:348] T 00000000000000000000000000000000 P c497e6ca916e4ab0b8aa706199f93e30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c497e6ca916e4ab0b8aa706199f93e30" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 44605 } }
I20260926 03:44:24.147226 6112 raft_consensus.cc:374] T 00000000000000000000000000000000 P c497e6ca916e4ab0b8aa706199f93e30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:44:24.147338 6112 raft_consensus.cc:729] T 00000000000000000000000000000000 P c497e6ca916e4ab0b8aa706199f93e30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c497e6ca916e4ab0b8aa706199f93e30, State: Initialized, Role: FOLLOWER
I20260926 03:44:24.147997 6112 consensus_queue.cc:260] T 00000000000000000000000000000000 P c497e6ca916e4ab0b8aa706199f93e30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c497e6ca916e4ab0b8aa706199f93e30" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 44605 } }
I20260926 03:44:24.148228 6112 raft_consensus.cc:388] T 00000000000000000000000000000000 P c497e6ca916e4ab0b8aa706199f93e30 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:44:24.148375 6112 raft_consensus.cc:482] T 00000000000000000000000000000000 P c497e6ca916e4ab0b8aa706199f93e30 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:44:24.148546 6112 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c497e6ca916e4ab0b8aa706199f93e30 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:44:24.151021 6112 raft_consensus.cc:504] T 00000000000000000000000000000000 P c497e6ca916e4ab0b8aa706199f93e30 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c497e6ca916e4ab0b8aa706199f93e30" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 44605 } }
I20260926 03:44:24.151458 6112 leader_election.cc:302] T 00000000000000000000000000000000 P c497e6ca916e4ab0b8aa706199f93e30 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c497e6ca916e4ab0b8aa706199f93e30; no voters:
I20260926 03:44:24.151945 6112 leader_election.cc:288] T 00000000000000000000000000000000 P c497e6ca916e4ab0b8aa706199f93e30 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:44:24.152100 6117 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c497e6ca916e4ab0b8aa706199f93e30 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:44:24.152791 6117 raft_consensus.cc:686] T 00000000000000000000000000000000 P c497e6ca916e4ab0b8aa706199f93e30 [term 1 LEADER]: Becoming Leader. State: Replica: c497e6ca916e4ab0b8aa706199f93e30, State: Running, Role: LEADER
I20260926 03:44:24.153278 6117 consensus_queue.cc:237] T 00000000000000000000000000000000 P c497e6ca916e4ab0b8aa706199f93e30 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c497e6ca916e4ab0b8aa706199f93e30" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 44605 } }
I20260926 03:44:24.154070 6112 sys_catalog.cc:564] T 00000000000000000000000000000000 P c497e6ca916e4ab0b8aa706199f93e30 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:44:24.156208 6119 sys_catalog.cc:455] T 00000000000000000000000000000000 P c497e6ca916e4ab0b8aa706199f93e30 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c497e6ca916e4ab0b8aa706199f93e30. Latest consensus state: current_term: 1 leader_uuid: "c497e6ca916e4ab0b8aa706199f93e30" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c497e6ca916e4ab0b8aa706199f93e30" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 44605 } } }
I20260926 03:44:24.156397 6118 sys_catalog.cc:455] T 00000000000000000000000000000000 P c497e6ca916e4ab0b8aa706199f93e30 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c497e6ca916e4ab0b8aa706199f93e30" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c497e6ca916e4ab0b8aa706199f93e30" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 44605 } } }
I20260926 03:44:24.156697 6118 sys_catalog.cc:458] T 00000000000000000000000000000000 P c497e6ca916e4ab0b8aa706199f93e30 [sys.catalog]: This master's current role is: LEADER
I20260926 03:44:24.156697 6119 sys_catalog.cc:458] T 00000000000000000000000000000000 P c497e6ca916e4ab0b8aa706199f93e30 [sys.catalog]: This master's current role is: LEADER
I20260926 03:44:24.158440 6127 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:44:24.163635 6127 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:44:24.171849 6127 catalog_manager.cc:1347] Generated new cluster ID: 6df7bfdf8f374e379397473eda17cfad
I20260926 03:44:24.171962 6127 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:44:24.195835 6127 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:44:24.197160 6127 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:44:24.206625 6127 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c497e6ca916e4ab0b8aa706199f93e30: Generated new TSK 0
I20260926 03:44:24.207765 6127 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260926 03:44:24.255312 6142 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260926 03:44:24.255450 6142 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260926 03:44:24.256453 6115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:24.228588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57394 (local address 127.0.194.254:44605)
0926 03:44:24.228793 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0926 03:44:24.228799 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:44:24.229339 (+ 540us) server_negotiation.cc:407] Connection header received
0926 03:44:24.229580 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:44:24.229584 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:44:24.229652 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:44:24.229779 (+ 127us) server_negotiation.cc:226] Negotiated authn=JWT
0926 03:44:24.230312 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:24.231078 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:24.233132 (+ 2054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:24.233571 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:24.233662 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:24.234248 (+ 586us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 03:44:24.255896 (+ 21648us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40175/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0926 03:44:24.256228 (+ 332us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:57394: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40175/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":57}
W20260926 03:44:24.256618 6115 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:57394: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40175/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260926 03:44:24.256860 6141 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:44:24.228383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:44605 (local address 127.0.0.1:57394)
0926 03:44:24.229217 (+ 834us) negotiation.cc:107] Waiting for socket to connect
0926 03:44:24.229242 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0926 03:44:24.229403 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:44:24.229826 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:44:24.229831 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:44:24.229845 (+ 14us) client_negotiation.cc:189] Negotiated authn=JWT
0926 03:44:24.230154 (+ 309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:24.230165 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:24.231214 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:24.231220 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:24.232995 (+ 1775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:24.233010 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:24.233735 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:24.233741 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:24.233936 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:24.233984 (+ 48us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 03:44:24.256162 (+ 22178us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40175/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0926 03:44:24.256469 (+ 307us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.194.254:44605: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40175/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":678,"thread_start_us":520,"threads_started":1}
I20260926 03:44:24.258719 779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu pid 6046
I20260926 03:44:24.259984 6107 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:44:24.352481 779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu with pid 6046
2026-09-26T03:44:24Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (746 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260926 03:44:24.407625 779 mini_oidc.cc:152] Starting JWKS server
I20260926 03:44:24.409380 779 webserver.cc:460] Webserver started at https://127.0.0.1:45047/ using document root <none> and password file <none>
I20260926 03:44:24.409595 779 mini_oidc.cc:189] Starting OIDC Discovery server
I20260926 03:44:24.410168 779 webserver.cc:460] Webserver started at http://127.0.0.1:38299/ using document root <none> and password file <none>
2026-09-26T03:44:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:44:24Z Disabled control of system clock
I20260926 03:44:24.437696 779 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
/tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790394221564982-779-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790394221564982-779-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790394221564982-779-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790394221564982-779-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.194.254:38327
--webserver_interface=127.0.194.254
--webserver_port=0
--builtin_ntp_servers=127.0.194.212:38453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.194.254:38327
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45047/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260926 03:44:24.761330 6149 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:44:24.761612 6149 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:44:24.761794 6149 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:44:24.768976 6149 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:44:24.769085 6149 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:44:24.769217 6149 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:44:24.769315 6149 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:44:24.769394 6149 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 03:44:24.769481 6149 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 03:44:24.769551 6149 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:45047/jwks.json
I20260926 03:44:24.779083 6149 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.194.212:38453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790394221564982-779-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790394221564982-779-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.194.254:38327
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.194.254:38327
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45047/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790394221564982-779-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.194.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790394221564982-779-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:44:24.780479 6149 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:44:24.782317 6149 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:44:24.794085 6155 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:44:24.794173 6157 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:44:24.794129 6154 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:44:24.795279 6149 server_base.cc:1034] running on GCE node
I20260926 03:44:24.796066 6149 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:44:24.797569 6149 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:44:24.798838 6149 hybrid_clock.cc:648] HybridClock initialized: now 1790394264798776 us; error 57 us; skew 500 ppm
I20260926 03:44:24.799201 6149 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 03:44:24.801538 6149 webserver.cc:460] Webserver started at http://127.0.194.254:34157/ using document root <none> and password file <none>
I20260926 03:44:24.802249 6149 fs_manager.cc:359] Metadata directory not provided
I20260926 03:44:24.802383 6149 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:44:24.802722 6149 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:44:24.805708 6149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790394221564982-779-0/minicluster-data/master-0/data/instance:
uuid: "7efe9621fb3c40d881d033d55267990c"
format_stamp: "Formatted at 2026-09-26 03:44:24 on dist-test-slave-n2th"
server_key: "4b7fab5f74a92294b3d388de9f5e0ef8"
server_key_iv: "f0da216e91a52360bcad653cb2661fb6"
server_key_version: "encryptionkey@0"
I20260926 03:44:24.806541 6149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790394221564982-779-0/minicluster-data/master-0/wal/instance:
uuid: "7efe9621fb3c40d881d033d55267990c"
format_stamp: "Formatted at 2026-09-26 03:44:24 on dist-test-slave-n2th"
server_key: "4b7fab5f74a92294b3d388de9f5e0ef8"
server_key_iv: "f0da216e91a52360bcad653cb2661fb6"
server_key_version: "encryptionkey@0"
I20260926 03:44:24.811965 6149 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260926 03:44:24.816169 6163 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:44:24.817796 6149 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 03:44:24.817983 6149 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790394221564982-779-0/minicluster-data/master-0/data,/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790394221564982-779-0/minicluster-data/master-0/wal
uuid: "7efe9621fb3c40d881d033d55267990c"
format_stamp: "Formatted at 2026-09-26 03:44:24 on dist-test-slave-n2th"
server_key: "4b7fab5f74a92294b3d388de9f5e0ef8"
server_key_iv: "f0da216e91a52360bcad653cb2661fb6"
server_key_version: "encryptionkey@0"
I20260926 03:44:24.818218 6149 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790394221564982-779-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790394221564982-779-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790394221564982-779-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:44:24.830313 6149 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:44:24.831296 6149 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:44:24.831600 6149 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:44:24.848949 6149 rpc_server.cc:307] RPC server started. Bound to: 127.0.194.254:38327
I20260926 03:44:24.848969 6214 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.194.254:38327 every 8 connection(s)
I20260926 03:44:24.850400 6149 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790394221564982-779-0/minicluster-data/master-0/data/info.pb
I20260926 03:44:24.854242 6215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:44:24.856581 779 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu as pid 6149
I20260926 03:44:24.856794 779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790394221564982-779-0/minicluster-data/master-0/wal/instance
I20260926 03:44:24.857424 779 external_mini_cluster.cc:1428] Setting key 615581755e8308be99f9a2f4b57424d2
I20260926 03:44:24.865861 6215 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7efe9621fb3c40d881d033d55267990c: Bootstrap starting.
I20260926 03:44:24.869231 6215 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7efe9621fb3c40d881d033d55267990c: Neither blocks nor log segments found. Creating new log.
I20260926 03:44:24.870400 6215 log.cc:824] T 00000000000000000000000000000000 P 7efe9621fb3c40d881d033d55267990c: Log is configured to *not* fsync() on all Append() calls
I20260926 03:44:24.871395 6218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:24.859495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52590 (local address 127.0.194.254:38327)
0926 03:44:24.860541 (+ 1046us) server_negotiation.cc:206] Beginning negotiation
0926 03:44:24.860552 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 03:44:24.860595 (+ 43us) server_negotiation.cc:407] Connection header received
0926 03:44:24.862371 (+ 1776us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:44:24.862395 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:44:24.862964 (+ 569us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:44:24.863384 (+ 420us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:44:24.863979 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:24.865245 (+ 1266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:24.866597 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:24.867376 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:24.867571 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:24.868733 (+ 1162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:44:24.868740 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:44:24.868754 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:44:24.869295 (+ 541us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:44:24.869416 (+ 121us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:44:24.869459 (+ 43us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:44:24.869588 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0926 03:44:24.869809 (+ 221us) server_negotiation.cc:299] Negotiation successful
0926 03:44:24.870107 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":627,"thread_start_us":382,"threads_started":1}
I20260926 03:44:24.873713 6215 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7efe9621fb3c40d881d033d55267990c: No bootstrap required, opened a new log
I20260926 03:44:24.875495 779 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 03:44:24.881822 6215 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7efe9621fb3c40d881d033d55267990c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7efe9621fb3c40d881d033d55267990c" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 38327 } }
I20260926 03:44:24.882217 6215 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7efe9621fb3c40d881d033d55267990c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:44:24.882333 6215 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7efe9621fb3c40d881d033d55267990c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7efe9621fb3c40d881d033d55267990c, State: Initialized, Role: FOLLOWER
I20260926 03:44:24.883031 6215 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7efe9621fb3c40d881d033d55267990c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7efe9621fb3c40d881d033d55267990c" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 38327 } }
I20260926 03:44:24.883309 6215 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7efe9621fb3c40d881d033d55267990c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:44:24.883412 6215 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7efe9621fb3c40d881d033d55267990c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:44:24.883533 6215 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7efe9621fb3c40d881d033d55267990c [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:44:24.885694 6215 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7efe9621fb3c40d881d033d55267990c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7efe9621fb3c40d881d033d55267990c" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 38327 } }
I20260926 03:44:24.886138 6215 leader_election.cc:302] T 00000000000000000000000000000000 P 7efe9621fb3c40d881d033d55267990c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7efe9621fb3c40d881d033d55267990c; no voters:
I20260926 03:44:24.886826 6215 leader_election.cc:288] T 00000000000000000000000000000000 P 7efe9621fb3c40d881d033d55267990c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:44:24.886920 6220 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7efe9621fb3c40d881d033d55267990c [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:44:24.887827 6220 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7efe9621fb3c40d881d033d55267990c [term 1 LEADER]: Becoming Leader. State: Replica: 7efe9621fb3c40d881d033d55267990c, State: Running, Role: LEADER
I20260926 03:44:24.888314 6220 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7efe9621fb3c40d881d033d55267990c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7efe9621fb3c40d881d033d55267990c" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 38327 } }
I20260926 03:44:24.888926 6215 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7efe9621fb3c40d881d033d55267990c [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:44:24.891053 6221 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7efe9621fb3c40d881d033d55267990c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7efe9621fb3c40d881d033d55267990c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7efe9621fb3c40d881d033d55267990c" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 38327 } } }
I20260926 03:44:24.891510 6221 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7efe9621fb3c40d881d033d55267990c [sys.catalog]: This master's current role is: LEADER
I20260926 03:44:24.893158 6222 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7efe9621fb3c40d881d033d55267990c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7efe9621fb3c40d881d033d55267990c. Latest consensus state: current_term: 1 leader_uuid: "7efe9621fb3c40d881d033d55267990c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7efe9621fb3c40d881d033d55267990c" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 38327 } } }
I20260926 03:44:24.893486 6222 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7efe9621fb3c40d881d033d55267990c [sys.catalog]: This master's current role is: LEADER
I20260926 03:44:24.893520 6229 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:44:24.898854 6229 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:44:24.908202 6229 catalog_manager.cc:1347] Generated new cluster ID: 2b63b9d6f0f4480fa4927192a3431589
I20260926 03:44:24.908345 6229 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:44:24.941918 6229 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:44:24.943343 6229 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:44:24.949682 6229 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7efe9621fb3c40d881d033d55267990c: Generated new TSK 0
I20260926 03:44:24.950461 6229 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 03:44:24.980100 6218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:24.966595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52600 (local address 127.0.194.254:38327)
0926 03:44:24.966875 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0926 03:44:24.966887 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 03:44:24.971460 (+ 4573us) server_negotiation.cc:407] Connection header received
0926 03:44:24.971802 (+ 342us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:44:24.971818 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:44:24.971941 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:44:24.972103 (+ 162us) server_negotiation.cc:226] Negotiated authn=JWT
0926 03:44:24.972662 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:24.973742 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:24.975708 (+ 1966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:24.976139 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:24.976294 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:24.976810 (+ 516us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 03:44:24.979486 (+ 2676us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45047/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0926 03:44:24.979817 (+ 331us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:52600: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45047/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":72}
W20260926 03:44:24.980219 6218 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:52600: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45047/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260926 03:44:24.982414 6244 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:44:24.966706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:38327 (local address 127.0.0.1:52600)
0926 03:44:24.971337 (+ 4631us) negotiation.cc:107] Waiting for socket to connect
0926 03:44:24.971361 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 03:44:24.971543 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:44:24.972119 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:44:24.972124 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:44:24.972137 (+ 13us) client_negotiation.cc:189] Negotiated authn=JWT
0926 03:44:24.972497 (+ 360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:24.972511 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:24.973889 (+ 1378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:24.973894 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:24.975526 (+ 1632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:24.975538 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:24.976345 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:24.976352 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:24.976508 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:24.976544 (+ 36us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 03:44:24.980652 (+ 4108us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45047/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0926 03:44:24.981871 (+ 1219us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.194.254:38327: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45047/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":4401,"mutex_wait_us":894,"thread_start_us":4238,"threads_started":1}
I20260926 03:44:24.983233 779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu pid 6149
I20260926 03:44:24.985769 6210 generic_service.cc:196] Checking for leaks (request via RPC)
E20260926 03:44:24.997175 6245 webserver.cc:533] Webserver: sslize failed:
E20260926 03:44:24.997318 6245 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260926 03:44:25.096662 779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu with pid 6149
2026-09-26T03:44:25Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (744 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-26T03:44:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:44:25Z Disabled control of system clock
I20260926 03:44:25.179148 779 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
/tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790394221564982-779-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790394221564982-779-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790394221564982-779-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790394221564982-779-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.194.254:38975
--webserver_interface=127.0.194.254
--webserver_port=0
--builtin_ntp_servers=127.0.194.212:45415
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.194.254:38975
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790394221564982-779-0/testchainca.cert with env {}
W20260926 03:44:25.506348 6250 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:44:25.506680 6250 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:44:25.506863 6250 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:44:25.514261 6250 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:44:25.514369 6250 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:44:25.514454 6250 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:44:25.514559 6250 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:44:25.514639 6250 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 03:44:25.514755 6250 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 03:44:25.514801 6250 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260926 03:44:25.524642 6250 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.194.212:45415
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790394221564982-779-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790394221564982-779-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.194.254:38975
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.194.254:38975
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790394221564982-779-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.194.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790394221564982-779-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790394221564982-779-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:44:25.525942 6250 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:44:25.527298 6250 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:44:25.538861 6255 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:44:25.538877 6258 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:44:25.538910 6256 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:44:25.539669 6250 server_base.cc:1034] running on GCE node
I20260926 03:44:25.540458 6250 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:44:25.541827 6250 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:44:25.543053 6250 hybrid_clock.cc:648] HybridClock initialized: now 1790394265542964 us; error 51 us; skew 500 ppm
I20260926 03:44:25.543404 6250 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 03:44:25.545684 6250 webserver.cc:460] Webserver started at http://127.0.194.254:43519/ using document root <none> and password file <none>
I20260926 03:44:25.546379 6250 fs_manager.cc:359] Metadata directory not provided
I20260926 03:44:25.546499 6250 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:44:25.546877 6250 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:44:25.550206 6250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790394221564982-779-0/minicluster-data/master-0/data/instance:
uuid: "0c80cd65a17c403887d842b4d6702d40"
format_stamp: "Formatted at 2026-09-26 03:44:25 on dist-test-slave-n2th"
server_key: "8598561a2f05c43567bc07512b5c26ad"
server_key_iv: "c3987b0a00a82fe5669cfdd827ce5c35"
server_key_version: "encryptionkey@0"
I20260926 03:44:25.551083 6250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790394221564982-779-0/minicluster-data/master-0/wal/instance:
uuid: "0c80cd65a17c403887d842b4d6702d40"
format_stamp: "Formatted at 2026-09-26 03:44:25 on dist-test-slave-n2th"
server_key: "8598561a2f05c43567bc07512b5c26ad"
server_key_iv: "c3987b0a00a82fe5669cfdd827ce5c35"
server_key_version: "encryptionkey@0"
I20260926 03:44:25.556051 6250 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260926 03:44:25.559414 6264 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:44:25.560731 6250 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 03:44:25.560918 6250 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790394221564982-779-0/minicluster-data/master-0/data,/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790394221564982-779-0/minicluster-data/master-0/wal
uuid: "0c80cd65a17c403887d842b4d6702d40"
format_stamp: "Formatted at 2026-09-26 03:44:25 on dist-test-slave-n2th"
server_key: "8598561a2f05c43567bc07512b5c26ad"
server_key_iv: "c3987b0a00a82fe5669cfdd827ce5c35"
server_key_version: "encryptionkey@0"
I20260926 03:44:25.561149 6250 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790394221564982-779-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790394221564982-779-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790394221564982-779-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:44:25.577312 6250 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:44:25.578166 6250 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:44:25.578455 6250 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:44:25.596405 6250 rpc_server.cc:307] RPC server started. Bound to: 127.0.194.254:38975
I20260926 03:44:25.596424 6315 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.194.254:38975 every 8 connection(s)
I20260926 03:44:25.597846 6250 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790394221564982-779-0/minicluster-data/master-0/data/info.pb
I20260926 03:44:25.598304 779 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu as pid 6250
I20260926 03:44:25.598490 779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790394221564982-779-0/minicluster-data/master-0/wal/instance
I20260926 03:44:25.599361 779 external_mini_cluster.cc:1428] Setting key afb27c30052fee1f4d962d7b01760c87
I20260926 03:44:25.602698 6316 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:44:25.613248 6319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:25.601331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58862 (local address 127.0.194.254:38975)
0926 03:44:25.602319 (+ 988us) server_negotiation.cc:206] Beginning negotiation
0926 03:44:25.602331 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 03:44:25.602373 (+ 42us) server_negotiation.cc:407] Connection header received
0926 03:44:25.604101 (+ 1728us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:44:25.604125 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:44:25.604605 (+ 480us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:44:25.605036 (+ 431us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:44:25.605647 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:25.607066 (+ 1419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:25.608382 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:25.609072 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:25.609279 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:25.610523 (+ 1244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:44:25.610530 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:44:25.610549 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:44:25.611293 (+ 744us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:44:25.611366 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:44:25.611391 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:44:25.611530 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0926 03:44:25.611784 (+ 254us) server_negotiation.cc:299] Negotiation successful
0926 03:44:25.612140 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":542,"thread_start_us":340,"threads_started":1}
I20260926 03:44:25.613999 6316 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c80cd65a17c403887d842b4d6702d40: Bootstrap starting.
I20260926 03:44:25.617192 779 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 03:44:25.617409 6316 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0c80cd65a17c403887d842b4d6702d40: Neither blocks nor log segments found. Creating new log.
I20260926 03:44:25.618398 6316 log.cc:824] T 00000000000000000000000000000000 P 0c80cd65a17c403887d842b4d6702d40: Log is configured to *not* fsync() on all Append() calls
I20260926 03:44:25.621469 6316 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c80cd65a17c403887d842b4d6702d40: No bootstrap required, opened a new log
I20260926 03:44:25.629441 6316 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0c80cd65a17c403887d842b4d6702d40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c80cd65a17c403887d842b4d6702d40" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 38975 } }
I20260926 03:44:25.629889 6316 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0c80cd65a17c403887d842b4d6702d40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:44:25.630002 6316 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0c80cd65a17c403887d842b4d6702d40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c80cd65a17c403887d842b4d6702d40, State: Initialized, Role: FOLLOWER
I20260926 03:44:25.630594 6316 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0c80cd65a17c403887d842b4d6702d40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c80cd65a17c403887d842b4d6702d40" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 38975 } }
I20260926 03:44:25.630908 6316 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0c80cd65a17c403887d842b4d6702d40 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:44:25.631053 6316 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0c80cd65a17c403887d842b4d6702d40 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:44:25.631234 6316 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0c80cd65a17c403887d842b4d6702d40 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:44:25.633520 6316 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0c80cd65a17c403887d842b4d6702d40 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c80cd65a17c403887d842b4d6702d40" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 38975 } }
I20260926 03:44:25.634056 6316 leader_election.cc:302] T 00000000000000000000000000000000 P 0c80cd65a17c403887d842b4d6702d40 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0c80cd65a17c403887d842b4d6702d40; no voters:
I20260926 03:44:25.634625 6316 leader_election.cc:288] T 00000000000000000000000000000000 P 0c80cd65a17c403887d842b4d6702d40 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:44:25.634819 6321 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0c80cd65a17c403887d842b4d6702d40 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:44:25.635593 6321 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0c80cd65a17c403887d842b4d6702d40 [term 1 LEADER]: Becoming Leader. State: Replica: 0c80cd65a17c403887d842b4d6702d40, State: Running, Role: LEADER
I20260926 03:44:25.636179 6321 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0c80cd65a17c403887d842b4d6702d40 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c80cd65a17c403887d842b4d6702d40" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 38975 } }
I20260926 03:44:25.636960 6316 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0c80cd65a17c403887d842b4d6702d40 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:44:25.639420 6322 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c80cd65a17c403887d842b4d6702d40 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0c80cd65a17c403887d842b4d6702d40" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c80cd65a17c403887d842b4d6702d40" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 38975 } } }
I20260926 03:44:25.639989 6322 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c80cd65a17c403887d842b4d6702d40 [sys.catalog]: This master's current role is: LEADER
I20260926 03:44:25.639505 6323 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c80cd65a17c403887d842b4d6702d40 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0c80cd65a17c403887d842b4d6702d40. Latest consensus state: current_term: 1 leader_uuid: "0c80cd65a17c403887d842b4d6702d40" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c80cd65a17c403887d842b4d6702d40" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 38975 } } }
I20260926 03:44:25.640633 6323 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c80cd65a17c403887d842b4d6702d40 [sys.catalog]: This master's current role is: LEADER
I20260926 03:44:25.641944 6330 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:44:25.647708 6330 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:44:25.656479 6330 catalog_manager.cc:1347] Generated new cluster ID: 1014e57e47e749788e9a1af23d31879b
I20260926 03:44:25.656617 6330 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:44:25.682207 6330 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:44:25.683583 6330 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:44:25.689669 6330 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0c80cd65a17c403887d842b4d6702d40: Generated new TSK 0
I20260926 03:44:25.690467 6330 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 03:44:26.177815 6319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:25.707880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58878 (local address 127.0.194.254:38975)
0926 03:44:25.708090 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0926 03:44:25.708097 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:44:25.708502 (+ 405us) server_negotiation.cc:407] Connection header received
0926 03:44:25.708733 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:44:25.708744 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:44:25.708852 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:44:25.709031 (+ 179us) server_negotiation.cc:226] Negotiated authn=JWT
0926 03:44:25.709434 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:25.710561 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:25.711993 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:25.712713 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:25.712859 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:25.713750 (+ 891us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 03:44:26.177276 (+463526us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0926 03:44:26.177602 (+ 326us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:58878: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":68}
W20260926 03:44:26.177932 6319 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:58878: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260926 03:44:26.178290 6345 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:44:25.707726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:38975 (local address 127.0.0.1:58878)
0926 03:44:25.708399 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0926 03:44:25.708415 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 03:44:25.708540 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:44:25.709043 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:44:25.709046 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:44:25.709056 (+ 10us) client_negotiation.cc:189] Negotiated authn=JWT
0926 03:44:25.709293 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:25.709300 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:25.710708 (+ 1408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:25.710711 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:25.711852 (+ 1141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:25.711863 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:25.713192 (+ 1329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:25.713199 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:25.713390 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:25.713438 (+ 48us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 03:44:26.177559 (+464121us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0926 03:44:26.177820 (+ 261us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.194.254:38975: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":395,"threads_started":1}
I20260926 03:44:26.180143 779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu pid 6250
I20260926 03:44:26.181147 6311 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:44:26.281407 779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu with pid 6250
2026-09-26T03:44:26Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (1166 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (410 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (420 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790394221564982-779-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:44:27 dist-test-slave-n2th krb5kdc[6360](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:44:27 dist-test-slave-n2th krb5kdc[6360](info): set up 2 sockets
Sep 26 03:44:27 dist-test-slave-n2th krb5kdc[6360](info): commencing operation
krb5kdc: starting...
W20260926 03:44:28.219538 779 mini_kdc.cc:121] Time spent starting KDC: real 1.069s user 0.004s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790394221564982-779-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790394221564982-779-0/krb5kdc/test-user.keytab.
Sep 26 03:44:28 dist-test-slave-n2th krb5kdc[6360](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394268, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:44:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:44:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.194.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.194.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.194.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790394221564982-779-0/krb5kdc/kudu_127.0.194.254.keytab.
Entry for principal kudu/127.0.194.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790394221564982-779-0/krb5kdc/kudu_127.0.194.254.keytab.
I20260926 03:44:28.423800 779 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
/tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790394221564982-779-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790394221564982-779-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790394221564982-779-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790394221564982-779-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.194.254:45501
--webserver_interface=127.0.194.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790394221564982-779-0/krb5kdc/kudu_127.0.194.254.keytab
--principal=kudu/127.0.194.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.194.212:38471
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.194.254:45501
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790394221564982-779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790394221564982-779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790394221564982-779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:44:28.744445 6373 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:44:28.744756 6373 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:44:28.744946 6373 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:44:28.752256 6373 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:44:28.752377 6373 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:44:28.752497 6373 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:44:28.752604 6373 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:44:28.752671 6373 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:44:28.762633 6373 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.194.212:38471
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790394221564982-779-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790394221564982-779-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.194.254:45501
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790394221564982-779-0/krb5kdc/kudu_127.0.194.254.keytab
--principal=kudu/127.0.194.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.194.254:45501
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790394221564982-779-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.194.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790394221564982-779-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:44:28.764021 6373 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:44:28.765372 6373 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:44:28.777164 6381 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:44:28.777218 6379 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:44:28.777263 6378 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:44:28.778084 6373 server_base.cc:1034] running on GCE node
I20260926 03:44:28.779019 6373 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:44:28.780459 6373 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:44:28.781663 6373 hybrid_clock.cc:648] HybridClock initialized: now 1790394268781607 us; error 50 us; skew 500 ppm
I20260926 03:44:28.782015 6373 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:44:28 dist-test-slave-n2th krb5kdc[6360](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394268, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:44:28.785102 6373 init.cc:377] Logged in from keytab as kudu/127.0.194.254@KRBTEST.COM (short username kudu)
I20260926 03:44:28.786577 6373 webserver.cc:460] Webserver started at http://127.0.194.254:43297/ using document root <none> and password file <none>
I20260926 03:44:28.787314 6373 fs_manager.cc:359] Metadata directory not provided
I20260926 03:44:28.787449 6373 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:44:28.787783 6373 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:44:28.790726 6373 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790394221564982-779-0/minicluster-data/master-0/data/instance:
uuid: "0ff2e40dbe794aa3830dc21a19b1261b"
format_stamp: "Formatted at 2026-09-26 03:44:28 on dist-test-slave-n2th"
server_key: "727def1295897b794e7f070e3af773c4"
server_key_iv: "73d35376238c01e5225236361e664488"
server_key_version: "encryptionkey@0"
I20260926 03:44:28.791585 6373 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790394221564982-779-0/minicluster-data/master-0/wal/instance:
uuid: "0ff2e40dbe794aa3830dc21a19b1261b"
format_stamp: "Formatted at 2026-09-26 03:44:28 on dist-test-slave-n2th"
server_key: "727def1295897b794e7f070e3af773c4"
server_key_iv: "73d35376238c01e5225236361e664488"
server_key_version: "encryptionkey@0"
I20260926 03:44:28.796840 6373 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 03:44:28.800344 6388 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:44:28.801766 6373 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 03:44:28.801993 6373 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790394221564982-779-0/minicluster-data/master-0/data,/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790394221564982-779-0/minicluster-data/master-0/wal
uuid: "0ff2e40dbe794aa3830dc21a19b1261b"
format_stamp: "Formatted at 2026-09-26 03:44:28 on dist-test-slave-n2th"
server_key: "727def1295897b794e7f070e3af773c4"
server_key_iv: "73d35376238c01e5225236361e664488"
server_key_version: "encryptionkey@0"
I20260926 03:44:28.802211 6373 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790394221564982-779-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790394221564982-779-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790394221564982-779-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:44:28.830513 6373 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:44:28.835081 6373 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:44:28.835341 6373 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:44:28.854023 6373 rpc_server.cc:307] RPC server started. Bound to: 127.0.194.254:45501
I20260926 03:44:28.854046 6439 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.194.254:45501 every 8 connection(s)
I20260926 03:44:28.855535 6373 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790394221564982-779-0/minicluster-data/master-0/data/info.pb
I20260926 03:44:28.859259 6440 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:44:28.863248 779 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu as pid 6373
I20260926 03:44:28.863469 779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790394221564982-779-0/minicluster-data/master-0/wal/instance
I20260926 03:44:28.864311 779 external_mini_cluster.cc:1428] Setting key 5857c538bfa3515364552d2410dd59ee
I20260926 03:44:28.873258 6440 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ff2e40dbe794aa3830dc21a19b1261b: Bootstrap starting.
I20260926 03:44:28.877431 6440 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0ff2e40dbe794aa3830dc21a19b1261b: Neither blocks nor log segments found. Creating new log.
I20260926 03:44:28.878918 6440 log.cc:824] T 00000000000000000000000000000000 P 0ff2e40dbe794aa3830dc21a19b1261b: Log is configured to *not* fsync() on all Append() calls
Sep 26 03:44:28 dist-test-slave-n2th krb5kdc[6360](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394268, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.194.254@KRBTEST.COM
I20260926 03:44:28.882004 6440 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ff2e40dbe794aa3830dc21a19b1261b: No bootstrap required, opened a new log
I20260926 03:44:28.888362 6443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:28.866457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60328 (local address 127.0.194.254:45501)
0926 03:44:28.867333 (+ 876us) server_negotiation.cc:206] Beginning negotiation
0926 03:44:28.867347 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 03:44:28.867389 (+ 42us) server_negotiation.cc:407] Connection header received
0926 03:44:28.868998 (+ 1609us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:44:28.869022 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:44:28.869478 (+ 456us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:44:28.869832 (+ 354us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:44:28.871664 (+ 1832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:28.873008 (+ 1344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:28.875054 (+ 2046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:28.875803 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:28.875962 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:28.880950 (+ 4988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:44:28.880958 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:44:28.880975 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:44:28.881012 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:44:28.884228 (+ 3216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:28.884988 (+ 760us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:28.884995 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:28.885002 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:28.885178 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:28.885558 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:28.885564 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:28.885567 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:28.886109 (+ 542us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:44:28.886427 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0926 03:44:28.886899 (+ 472us) server_negotiation.cc:299] Negotiation successful
0926 03:44:28.887330 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":656,"thread_start_us":435,"threads_started":1}
I20260926 03:44:28.890141 6440 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0ff2e40dbe794aa3830dc21a19b1261b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ff2e40dbe794aa3830dc21a19b1261b" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 45501 } }
I20260926 03:44:28.890588 6440 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0ff2e40dbe794aa3830dc21a19b1261b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:44:28.890697 6440 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0ff2e40dbe794aa3830dc21a19b1261b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ff2e40dbe794aa3830dc21a19b1261b, State: Initialized, Role: FOLLOWER
I20260926 03:44:28.891390 6440 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0ff2e40dbe794aa3830dc21a19b1261b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ff2e40dbe794aa3830dc21a19b1261b" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 45501 } }
I20260926 03:44:28.891665 6440 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0ff2e40dbe794aa3830dc21a19b1261b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:44:28.891774 6440 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0ff2e40dbe794aa3830dc21a19b1261b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:44:28.891892 6440 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0ff2e40dbe794aa3830dc21a19b1261b [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:44:28.894112 6440 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0ff2e40dbe794aa3830dc21a19b1261b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ff2e40dbe794aa3830dc21a19b1261b" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 45501 } }
I20260926 03:44:28.894786 6440 leader_election.cc:302] T 00000000000000000000000000000000 P 0ff2e40dbe794aa3830dc21a19b1261b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0ff2e40dbe794aa3830dc21a19b1261b; no voters:
I20260926 03:44:28.895397 6440 leader_election.cc:288] T 00000000000000000000000000000000 P 0ff2e40dbe794aa3830dc21a19b1261b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:44:28.895540 6445 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0ff2e40dbe794aa3830dc21a19b1261b [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:44:28.896245 6445 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0ff2e40dbe794aa3830dc21a19b1261b [term 1 LEADER]: Becoming Leader. State: Replica: 0ff2e40dbe794aa3830dc21a19b1261b, State: Running, Role: LEADER
I20260926 03:44:28.896842 6445 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0ff2e40dbe794aa3830dc21a19b1261b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ff2e40dbe794aa3830dc21a19b1261b" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 45501 } }
I20260926 03:44:28.897488 6440 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0ff2e40dbe794aa3830dc21a19b1261b [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:44:28.900053 6447 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ff2e40dbe794aa3830dc21a19b1261b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0ff2e40dbe794aa3830dc21a19b1261b. Latest consensus state: current_term: 1 leader_uuid: "0ff2e40dbe794aa3830dc21a19b1261b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ff2e40dbe794aa3830dc21a19b1261b" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 45501 } } }
I20260926 03:44:28.900506 6447 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ff2e40dbe794aa3830dc21a19b1261b [sys.catalog]: This master's current role is: LEADER
I20260926 03:44:28.900862 6446 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ff2e40dbe794aa3830dc21a19b1261b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0ff2e40dbe794aa3830dc21a19b1261b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ff2e40dbe794aa3830dc21a19b1261b" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 45501 } } }
I20260926 03:44:28.901639 6446 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ff2e40dbe794aa3830dc21a19b1261b [sys.catalog]: This master's current role is: LEADER
I20260926 03:44:28.903280 6454 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:44:28.908609 6454 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:44:28.916982 6454 catalog_manager.cc:1347] Generated new cluster ID: f3af5add65424f3d9c61dde4a582c28c
I20260926 03:44:28.917110 6454 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:44:28.929534 6454 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:44:28.930977 6454 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:44:28.942440 6454 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0ff2e40dbe794aa3830dc21a19b1261b: Generated new TSK 0
I20260926 03:44:28.943421 6454 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.194.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.194.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.194.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790394221564982-779-0/krb5kdc/kudu_127.0.194.193.keytab.
Entry for principal kudu/127.0.194.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790394221564982-779-0/krb5kdc/kudu_127.0.194.193.keytab.
I20260926 03:44:29.010227 779 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
/tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790394221564982-779-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790394221564982-779-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790394221564982-779-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790394221564982-779-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.194.193:0
--local_ip_for_outbound_sockets=127.0.194.193
--webserver_interface=127.0.194.193
--webserver_port=0
--tserver_master_addrs=127.0.194.254:45501
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790394221564982-779-0/krb5kdc/kudu_127.0.194.193.keytab
--principal=kudu/127.0.194.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.194.212:38471
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790394221564982-779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790394221564982-779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790394221564982-779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:44:29.334239 6466 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:44:29.334581 6466 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:44:29.334802 6466 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:44:29.342216 6466 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:44:29.342378 6466 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:44:29.342554 6466 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.194.193
I20260926 03:44:29.352473 6466 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.194.212:38471
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790394221564982-779-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790394221564982-779-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790394221564982-779-0/krb5kdc/kudu_127.0.194.193.keytab
--principal=kudu/127.0.194.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.194.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790394221564982-779-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.194.193
--webserver_port=0
--tserver_master_addrs=127.0.194.254:45501
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.194.193
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790394221564982-779-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:44:29.353786 6466 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:44:29.355173 6466 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:44:29.368093 6474 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:44:29.368196 6471 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:44:29.368132 6472 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:44:29.369004 6466 server_base.cc:1034] running on GCE node
I20260926 03:44:29.369611 6466 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:44:29.370556 6466 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:44:29.371802 6466 hybrid_clock.cc:648] HybridClock initialized: now 1790394269371756 us; error 55 us; skew 500 ppm
I20260926 03:44:29.372161 6466 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:44:29 dist-test-slave-n2th krb5kdc[6360](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394269, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:44:29.379216 6466 init.cc:377] Logged in from keytab as kudu/127.0.194.193@KRBTEST.COM (short username kudu)
I20260926 03:44:29.380695 6466 webserver.cc:460] Webserver started at http://127.0.194.193:34795/ using document root <none> and password file <none>
I20260926 03:44:29.381438 6466 fs_manager.cc:359] Metadata directory not provided
I20260926 03:44:29.381543 6466 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:44:29.381806 6466 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:44:29.384742 6466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790394221564982-779-0/minicluster-data/ts-0/data/instance:
uuid: "9324bd3b5187411ca10d9f97f5db7878"
format_stamp: "Formatted at 2026-09-26 03:44:29 on dist-test-slave-n2th"
server_key: "e344c72b15fd693cdaea3df57c97bdc1"
server_key_iv: "d219ea374441cd45a9e377f939040483"
server_key_version: "encryptionkey@0"
I20260926 03:44:29.385481 6466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790394221564982-779-0/minicluster-data/ts-0/wal/instance:
uuid: "9324bd3b5187411ca10d9f97f5db7878"
format_stamp: "Formatted at 2026-09-26 03:44:29 on dist-test-slave-n2th"
server_key: "e344c72b15fd693cdaea3df57c97bdc1"
server_key_iv: "d219ea374441cd45a9e377f939040483"
server_key_version: "encryptionkey@0"
I20260926 03:44:29.390933 6466 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260926 03:44:29.394300 6481 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:44:29.395714 6466 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 03:44:29.395857 6466 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790394221564982-779-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790394221564982-779-0/minicluster-data/ts-0/wal
uuid: "9324bd3b5187411ca10d9f97f5db7878"
format_stamp: "Formatted at 2026-09-26 03:44:29 on dist-test-slave-n2th"
server_key: "e344c72b15fd693cdaea3df57c97bdc1"
server_key_iv: "d219ea374441cd45a9e377f939040483"
server_key_version: "encryptionkey@0"
I20260926 03:44:29.396026 6466 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790394221564982-779-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790394221564982-779-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790394221564982-779-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:44:29.417704 6466 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:44:29.422278 6466 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:44:29.422520 6466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:44:29.423506 6466 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:44:29.424803 6466 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:44:29.424889 6466 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:44:29.424984 6466 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:44:29.425040 6466 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:44:29.457247 6466 rpc_server.cc:307] RPC server started. Bound to: 127.0.194.193:37847
I20260926 03:44:29.457326 6593 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.194.193:37847 every 8 connection(s)
I20260926 03:44:29.458547 6466 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790394221564982-779-0/minicluster-data/ts-0/data/info.pb
I20260926 03:44:29.462867 779 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu as pid 6466
I20260926 03:44:29.463061 779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790394221564982-779-0/minicluster-data/ts-0/wal/instance
I20260926 03:44:29.463675 779 external_mini_cluster.cc:1428] Setting key c96eed013fd74316f0c017df56bd97eb
Sep 26 03:44:29 dist-test-slave-n2th krb5kdc[6360](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394269, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.193@KRBTEST.COM for kudu/127.0.194.254@KRBTEST.COM
I20260926 03:44:29.479812 6596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:29.461009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.193:37983 (local address 127.0.194.254:45501)
0926 03:44:29.461453 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0926 03:44:29.461459 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:44:29.462425 (+ 966us) server_negotiation.cc:407] Connection header received
0926 03:44:29.464471 (+ 2046us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:44:29.464476 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:44:29.464555 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:44:29.464680 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:44:29.466451 (+ 1771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:29.467688 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:29.469217 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:29.469933 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:29.470074 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:29.473966 (+ 3892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:44:29.473971 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:44:29.473974 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:44:29.474007 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:44:29.476647 (+ 2640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:29.477435 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:29.477441 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:29.477444 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:29.477567 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:29.477986 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:29.477993 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:29.477997 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:29.478261 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:44:29.478411 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0926 03:44:29.479202 (+ 791us) server_negotiation.cc:299] Negotiation successful
0926 03:44:29.479435 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":216,"threads_started":1}
I20260926 03:44:29.480803 6597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:29.461588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:45501 (local address 127.0.194.193:37983)
0926 03:44:29.462270 (+ 682us) negotiation.cc:107] Waiting for socket to connect
0926 03:44:29.462308 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 03:44:29.464140 (+ 1832us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:44:29.464943 (+ 803us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:44:29.464954 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:44:29.465699 (+ 745us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:44:29.466267 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:29.466285 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:29.467852 (+ 1567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:29.467857 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:29.469078 (+ 1221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:29.469089 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:29.470143 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:29.470147 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:29.470354 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:29.471084 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:44:29.471110 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:44:29.473731 (+ 2621us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:44:29.476820 (+ 3089us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:44:29.476829 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:44:29.476848 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:44:29.477304 (+ 456us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:44:29.477676 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:44:29.477682 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:44:29.477685 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:44:29.477877 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:44:29.478384 (+ 507us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:44:29.478393 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:44:29.478815 (+ 422us) client_negotiation.cc:769] Sending connection context
0926 03:44:29.479095 (+ 280us) client_negotiation.cc:240] Negotiation successful
0926 03:44:29.479664 (+ 569us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":267,"threads_started":1}
I20260926 03:44:29.482573 6594 heartbeater.cc:347] Connected to a master server at 127.0.194.254:45501
I20260926 03:44:29.482905 6594 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.194.194@KRBTEST.COM; defaulting to no policy
I20260926 03:44:29.483630 6594 heartbeater.cc:511] Master 127.0.194.254:45501 requested a full tablet report, sending...
I20260926 03:44:29.485723 6405 ts_manager.cc:194] Registered new tserver with Master: 9324bd3b5187411ca10d9f97f5db7878 (127.0.194.193:37847)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.194.194@KRBTEST.COM" created.
I20260926 03:44:29.487221 6405 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.194.193@KRBTEST.COM'} at 127.0.194.193:37983
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.194.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790394221564982-779-0/krb5kdc/kudu_127.0.194.194.keytab.
Entry for principal kudu/127.0.194.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790394221564982-779-0/krb5kdc/kudu_127.0.194.194.keytab.
I20260926 03:44:29.509538 779 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
/tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790394221564982-779-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790394221564982-779-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790394221564982-779-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790394221564982-779-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.194.194:0
--local_ip_for_outbound_sockets=127.0.194.194
--webserver_interface=127.0.194.194
--webserver_port=0
--tserver_master_addrs=127.0.194.254:45501
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790394221564982-779-0/krb5kdc/kudu_127.0.194.194.keytab
--principal=kudu/127.0.194.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.194.212:38471
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790394221564982-779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790394221564982-779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790394221564982-779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:44:29.844038 6600 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:44:29.844344 6600 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:44:29.844524 6600 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:44:29.851866 6600 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:44:29.851977 6600 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:44:29.852183 6600 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.194.194
I20260926 03:44:29.862237 6600 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.194.212:38471
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790394221564982-779-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790394221564982-779-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790394221564982-779-0/krb5kdc/kudu_127.0.194.194.keytab
--principal=kudu/127.0.194.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.194.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790394221564982-779-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.194.194
--webserver_port=0
--tserver_master_addrs=127.0.194.254:45501
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.194.194
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790394221564982-779-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:44:29.863591 6600 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:44:29.864921 6600 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:44:29.878333 6606 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:44:29.878237 6608 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:44:29.878343 6605 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:44:29.879128 6600 server_base.cc:1034] running on GCE node
I20260926 03:44:29.879767 6600 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:44:29.880652 6600 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:44:29.881842 6600 hybrid_clock.cc:648] HybridClock initialized: now 1790394269881796 us; error 38 us; skew 500 ppm
I20260926 03:44:29.882160 6600 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:44:29 dist-test-slave-n2th krb5kdc[6360](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394269, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:44:29.885340 6600 init.cc:377] Logged in from keytab as kudu/127.0.194.194@KRBTEST.COM (short username kudu)
I20260926 03:44:29.886677 6600 webserver.cc:460] Webserver started at http://127.0.194.194:41197/ using document root <none> and password file <none>
I20260926 03:44:29.887416 6600 fs_manager.cc:359] Metadata directory not provided
I20260926 03:44:29.887549 6600 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:44:29.887863 6600 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:44:29.890837 6600 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790394221564982-779-0/minicluster-data/ts-1/data/instance:
uuid: "904285498f4540a2bec10f5e8ea9138c"
format_stamp: "Formatted at 2026-09-26 03:44:29 on dist-test-slave-n2th"
server_key: "ace25f8539070db3ac58f3b9621f1cbd"
server_key_iv: "5c1248d30e72bc0ff81942fbaa9892d6"
server_key_version: "encryptionkey@0"
I20260926 03:44:29.891605 6600 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790394221564982-779-0/minicluster-data/ts-1/wal/instance:
uuid: "904285498f4540a2bec10f5e8ea9138c"
format_stamp: "Formatted at 2026-09-26 03:44:29 on dist-test-slave-n2th"
server_key: "ace25f8539070db3ac58f3b9621f1cbd"
server_key_iv: "5c1248d30e72bc0ff81942fbaa9892d6"
server_key_version: "encryptionkey@0"
I20260926 03:44:29.896633 6600 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260926 03:44:29.900349 6615 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:44:29.901886 6600 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 03:44:29.902063 6600 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790394221564982-779-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790394221564982-779-0/minicluster-data/ts-1/wal
uuid: "904285498f4540a2bec10f5e8ea9138c"
format_stamp: "Formatted at 2026-09-26 03:44:29 on dist-test-slave-n2th"
server_key: "ace25f8539070db3ac58f3b9621f1cbd"
server_key_iv: "5c1248d30e72bc0ff81942fbaa9892d6"
server_key_version: "encryptionkey@0"
I20260926 03:44:29.902278 6600 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790394221564982-779-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790394221564982-779-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790394221564982-779-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:44:29.914670 6600 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:44:29.919122 6600 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:44:29.919358 6600 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:44:29.920212 6600 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:44:29.921440 6600 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:44:29.921511 6600 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:44:29.921593 6600 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:44:29.921636 6600 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:44:29.954195 6600 rpc_server.cc:307] RPC server started. Bound to: 127.0.194.194:35259
I20260926 03:44:29.954238 6727 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.194.194:35259 every 8 connection(s)
I20260926 03:44:29.955595 6600 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790394221564982-779-0/minicluster-data/ts-1/data/info.pb
I20260926 03:44:29.959723 779 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu as pid 6600
I20260926 03:44:29.959896 779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790394221564982-779-0/minicluster-data/ts-1/wal/instance
I20260926 03:44:29.960340 779 external_mini_cluster.cc:1428] Setting key 86c875af132d27998672d99348353697
Sep 26 03:44:29 dist-test-slave-n2th krb5kdc[6360](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394269, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.194@KRBTEST.COM for kudu/127.0.194.254@KRBTEST.COM
I20260926 03:44:29.976636 6596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:29.958139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.194:37755 (local address 127.0.194.254:45501)
0926 03:44:29.958315 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0926 03:44:29.958319 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:44:29.959537 (+ 1218us) server_negotiation.cc:407] Connection header received
0926 03:44:29.961627 (+ 2090us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:44:29.961632 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:44:29.961718 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:44:29.961827 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:44:29.963463 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:29.964169 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:29.965707 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:29.966146 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:29.966284 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:29.969750 (+ 3466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:44:29.969759 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:44:29.969767 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:44:29.969812 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:44:29.973352 (+ 3540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:29.974117 (+ 765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:29.974124 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:29.974131 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:29.974282 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:29.974674 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:29.974681 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:29.974688 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:29.975097 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:44:29.975251 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0926 03:44:29.975898 (+ 647us) server_negotiation.cc:299] Negotiation successful
0926 03:44:29.976177 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 03:44:29.977286 6730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:29.958595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:45501 (local address 127.0.194.194:37755)
0926 03:44:29.959377 (+ 782us) negotiation.cc:107] Waiting for socket to connect
0926 03:44:29.959421 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 03:44:29.961358 (+ 1937us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:44:29.962084 (+ 726us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:44:29.962096 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:44:29.962655 (+ 559us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:44:29.963288 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:29.963306 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:29.964312 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:29.964323 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:29.965563 (+ 1240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:29.965574 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:29.966376 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:29.966383 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:29.966605 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:29.967357 (+ 752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:44:29.967379 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:44:29.969534 (+ 2155us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:44:29.973500 (+ 3966us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:44:29.973511 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:44:29.973529 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:44:29.973981 (+ 452us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:44:29.974388 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:44:29.974392 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:44:29.974394 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:44:29.974561 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:44:29.975211 (+ 650us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:44:29.975217 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:44:29.975526 (+ 309us) client_negotiation.cc:769] Sending connection context
0926 03:44:29.975743 (+ 217us) client_negotiation.cc:240] Negotiation successful
0926 03:44:29.976138 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":334,"threads_started":1}
I20260926 03:44:29.978823 6728 heartbeater.cc:347] Connected to a master server at 127.0.194.254:45501
I20260926 03:44:29.979122 6728 heartbeater.cc:464] Registering TS with master...
I20260926 03:44:29.979811 6728 heartbeater.cc:511] Master 127.0.194.254:45501 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.194.195@KRBTEST.COM; defaulting to no policy
I20260926 03:44:29.981254 6405 ts_manager.cc:194] Registered new tserver with Master: 904285498f4540a2bec10f5e8ea9138c (127.0.194.194:35259)
I20260926 03:44:29.982069 6405 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.194.194@KRBTEST.COM'} at 127.0.194.194:37755
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.194.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.194.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790394221564982-779-0/krb5kdc/kudu_127.0.194.195.keytab.
Entry for principal kudu/127.0.194.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790394221564982-779-0/krb5kdc/kudu_127.0.194.195.keytab.
I20260926 03:44:30.007915 779 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
/tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790394221564982-779-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790394221564982-779-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790394221564982-779-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790394221564982-779-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.194.195:0
--local_ip_for_outbound_sockets=127.0.194.195
--webserver_interface=127.0.194.195
--webserver_port=0
--tserver_master_addrs=127.0.194.254:45501
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790394221564982-779-0/krb5kdc/kudu_127.0.194.195.keytab
--principal=kudu/127.0.194.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.194.212:38471
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790394221564982-779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790394221564982-779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790394221564982-779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:44:30.328704 6733 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:44:30.329006 6733 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:44:30.329173 6733 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:44:30.336584 6733 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:44:30.336699 6733 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:44:30.336928 6733 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.194.195
I20260926 03:44:30.346925 6733 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.194.212:38471
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790394221564982-779-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790394221564982-779-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790394221564982-779-0/krb5kdc/kudu_127.0.194.195.keytab
--principal=kudu/127.0.194.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.194.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790394221564982-779-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.194.195
--webserver_port=0
--tserver_master_addrs=127.0.194.254:45501
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.194.195
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790394221564982-779-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:44:30.348208 6733 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:44:30.349543 6733 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:44:30.362324 6738 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:44:30.362396 6739 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:44:30.362404 6741 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:44:30.363250 6733 server_base.cc:1034] running on GCE node
I20260926 03:44:30.363837 6733 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:44:30.364756 6733 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:44:30.365995 6733 hybrid_clock.cc:648] HybridClock initialized: now 1790394270365948 us; error 40 us; skew 500 ppm
I20260926 03:44:30.366304 6733 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:44:30 dist-test-slave-n2th krb5kdc[6360](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394270, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:44:30.369338 6733 init.cc:377] Logged in from keytab as kudu/127.0.194.195@KRBTEST.COM (short username kudu)
I20260926 03:44:30.370877 6733 webserver.cc:460] Webserver started at http://127.0.194.195:45469/ using document root <none> and password file <none>
I20260926 03:44:30.371549 6733 fs_manager.cc:359] Metadata directory not provided
I20260926 03:44:30.371691 6733 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:44:30.371989 6733 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:44:30.374873 6733 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790394221564982-779-0/minicluster-data/ts-2/data/instance:
uuid: "cdea5a10dd2249798e544ac2a3dd6565"
format_stamp: "Formatted at 2026-09-26 03:44:30 on dist-test-slave-n2th"
server_key: "a389e08f104ef77d13b299931767aa0b"
server_key_iv: "9063ecba693f34aa11658d3d84155a2a"
server_key_version: "encryptionkey@0"
I20260926 03:44:30.375658 6733 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790394221564982-779-0/minicluster-data/ts-2/wal/instance:
uuid: "cdea5a10dd2249798e544ac2a3dd6565"
format_stamp: "Formatted at 2026-09-26 03:44:30 on dist-test-slave-n2th"
server_key: "a389e08f104ef77d13b299931767aa0b"
server_key_iv: "9063ecba693f34aa11658d3d84155a2a"
server_key_version: "encryptionkey@0"
I20260926 03:44:30.380673 6733 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 03:44:30.383970 6748 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:44:30.385268 6733 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 03:44:30.385447 6733 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790394221564982-779-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790394221564982-779-0/minicluster-data/ts-2/wal
uuid: "cdea5a10dd2249798e544ac2a3dd6565"
format_stamp: "Formatted at 2026-09-26 03:44:30 on dist-test-slave-n2th"
server_key: "a389e08f104ef77d13b299931767aa0b"
server_key_iv: "9063ecba693f34aa11658d3d84155a2a"
server_key_version: "encryptionkey@0"
I20260926 03:44:30.385643 6733 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790394221564982-779-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790394221564982-779-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790394221564982-779-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:44:30.397941 6733 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:44:30.402136 6733 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:44:30.402434 6733 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:44:30.403527 6733 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:44:30.404980 6733 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:44:30.405088 6733 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:44:30.405237 6733 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:44:30.405320 6733 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:44:30.438242 6733 rpc_server.cc:307] RPC server started. Bound to: 127.0.194.195:38939
I20260926 03:44:30.438297 6860 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.194.195:38939 every 8 connection(s)
I20260926 03:44:30.439553 6733 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790394221564982-779-0/minicluster-data/ts-2/data/info.pb
I20260926 03:44:30.447505 779 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu as pid 6733
I20260926 03:44:30.447732 779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790394221564982-779-0/minicluster-data/ts-2/wal/instance
I20260926 03:44:30.448380 779 external_mini_cluster.cc:1428] Setting key 89a3caa53a64dd573998b3b93d4d8021
Sep 26 03:44:30 dist-test-slave-n2th krb5kdc[6360](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394270, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.195@KRBTEST.COM for kudu/127.0.194.254@KRBTEST.COM
I20260926 03:44:30.460124 6596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:30.442166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.195:37835 (local address 127.0.194.254:45501)
0926 03:44:30.442401 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0926 03:44:30.442408 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:44:30.443638 (+ 1230us) server_negotiation.cc:407] Connection header received
0926 03:44:30.445350 (+ 1712us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:44:30.445354 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:44:30.445438 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:44:30.445530 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:44:30.447054 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:30.447894 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:30.450390 (+ 2496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:30.450897 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:30.451035 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:30.454536 (+ 3501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:44:30.454541 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:44:30.454545 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:44:30.454577 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:44:30.457128 (+ 2551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:30.457844 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:30.457851 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:30.457855 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:30.457986 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:30.458418 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:30.458426 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:30.458430 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:30.458692 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:44:30.458904 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0926 03:44:30.459568 (+ 664us) server_negotiation.cc:299] Negotiation successful
0926 03:44:30.459785 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 03:44:30.461167 6863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:30.442772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:45501 (local address 127.0.194.195:37835)
0926 03:44:30.443474 (+ 702us) negotiation.cc:107] Waiting for socket to connect
0926 03:44:30.443516 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 03:44:30.445094 (+ 1578us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:44:30.445727 (+ 633us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:44:30.445736 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:44:30.446252 (+ 516us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:44:30.446896 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:30.446916 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:30.448057 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:30.448064 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:30.450218 (+ 2154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:30.450238 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:30.451121 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:30.451131 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:30.451383 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:30.452163 (+ 780us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:44:30.452184 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:44:30.454346 (+ 2162us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:44:30.457266 (+ 2920us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:44:30.457275 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:44:30.457292 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:44:30.457711 (+ 419us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:44:30.458110 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:44:30.458115 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:44:30.458118 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:44:30.458301 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:44:30.458889 (+ 588us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:44:30.458896 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:44:30.459209 (+ 313us) client_negotiation.cc:769] Sending connection context
0926 03:44:30.459476 (+ 267us) client_negotiation.cc:240] Negotiation successful
0926 03:44:30.459997 (+ 521us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":264,"threads_started":1}
I20260926 03:44:30.462556 6861 heartbeater.cc:347] Connected to a master server at 127.0.194.254:45501
I20260926 03:44:30.462904 6861 heartbeater.cc:464] Registering TS with master...
I20260926 03:44:30.463766 6861 heartbeater.cc:511] Master 127.0.194.254:45501 requested a full tablet report, sending...
I20260926 03:44:30.465197 6405 ts_manager.cc:194] Registered new tserver with Master: cdea5a10dd2249798e544ac2a3dd6565 (127.0.194.195:38939)
I20260926 03:44:30.465988 6405 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.194.195@KRBTEST.COM'} at 127.0.194.195:37835
I20260926 03:44:30.476439 779 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 03:44:30.490059 6594 heartbeater.cc:503] Master 127.0.194.254:45501 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790394221564982-779-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790394221564982-779-0/krb5kdc/test-admin.keytab.
Sep 26 03:44:30 dist-test-slave-n2th krb5kdc[6360](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394270, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:44:30.504274 779 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260926 03:44:30.517262 6868 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:44:30.507282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48208 (local address 127.0.194.193:37847)
0926 03:44:30.515382 (+ 8100us) server_negotiation.cc:206] Beginning negotiation
0926 03:44:30.515390 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:44:30.515419 (+ 29us) server_negotiation.cc:407] Connection header received
0926 03:44:30.515482 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:44:30.515489 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:44:30.515707 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:44:30.515855 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:44:30.516787 (+ 932us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48208: BlockingRecv error: recv got EOF from 127.0.0.1:48208 (error 108)
Metrics: {"server-negotiator.queue_time_us":7916,"thread_start_us":7720,"threads_started":1}
W20260926 03:44:30.517493 779 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790394221564982-779-0/krb5kdc/krb5cc' of type FILE
Sep 26 03:44:30 dist-test-slave-n2th krb5kdc[6360](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394270, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:44:30.519269 779 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 26 03:44:30 dist-test-slave-n2th krb5kdc[6360](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394270, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:44:30.520643 779 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 26 03:44:30 dist-test-slave-n2th krb5kdc[6360](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394270, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.194.193@KRBTEST.COM
I20260926 03:44:30.535840 6868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:30.522212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48218 (local address 127.0.194.193:37847)
0926 03:44:30.522381 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0926 03:44:30.522385 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:44:30.522802 (+ 417us) server_negotiation.cc:407] Connection header received
0926 03:44:30.522983 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:44:30.522987 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:44:30.523049 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:44:30.523132 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:44:30.524254 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:30.525216 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:30.526388 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:30.527021 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:30.527123 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:30.530717 (+ 3594us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:44:30.530770 (+ 53us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:44:30.530805 (+ 35us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:44:30.530849 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:44:30.533449 (+ 2600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:30.534054 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:30.534061 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:30.534064 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:30.534172 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:30.534494 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:30.534497 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:30.534499 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:30.534901 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:44:30.535111 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0926 03:44:30.535382 (+ 271us) server_negotiation.cc:299] Negotiation successful
0926 03:44:30.535573 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
W20260926 03:44:30.541414 6868 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:44:30.539666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48226 (local address 127.0.194.193:37847)
0926 03:44:30.539861 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0926 03:44:30.539868 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:44:30.540148 (+ 280us) server_negotiation.cc:407] Connection header received
0926 03:44:30.540327 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:44:30.540336 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:44:30.540399 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:44:30.540495 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:44:30.541176 (+ 681us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48226: BlockingRecv error: recv got EOF from 127.0.0.1:48226 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260926 03:44:30.541787 779 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790394221564982-779-0/krb5kdc/krb5cc' of type FILE
Sep 26 03:44:30 dist-test-slave-n2th krb5kdc[6360](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394270, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:44:30.543330 779 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 26 03:44:30 dist-test-slave-n2th krb5kdc[6360](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394270, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:44:30.544592 779 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 26 03:44:30 dist-test-slave-n2th krb5kdc[6360](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394270, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.194.193@KRBTEST.COM
I20260926 03:44:30.557785 6868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:30.545910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48230 (local address 127.0.194.193:37847)
0926 03:44:30.546100 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0926 03:44:30.546104 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:44:30.546225 (+ 121us) server_negotiation.cc:407] Connection header received
0926 03:44:30.546338 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:44:30.546342 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:44:30.546402 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:44:30.546476 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:44:30.547379 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:30.548035 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:30.549181 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:30.549646 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:30.549745 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:30.553090 (+ 3345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:44:30.553095 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:44:30.553098 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:44:30.553140 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:44:30.555506 (+ 2366us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:30.556099 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:30.556107 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:30.556113 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:30.556265 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:30.556605 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:30.556610 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:30.556612 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:30.556920 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:44:30.557109 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0926 03:44:30.557321 (+ 212us) server_negotiation.cc:299] Negotiation successful
0926 03:44:30.557506 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
W20260926 03:44:30.561955 6868 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:44:30.560295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48238 (local address 127.0.194.193:37847)
0926 03:44:30.560500 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0926 03:44:30.560506 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:44:30.560729 (+ 223us) server_negotiation.cc:407] Connection header received
0926 03:44:30.560919 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:44:30.560923 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:44:30.560987 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:44:30.561081 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:44:30.561737 (+ 656us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48238: BlockingRecv error: recv got EOF from 127.0.0.1:48238 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
W20260926 03:44:30.562140 779 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790394221564982-779-0/krb5kdc/krb5cc' of type FILE
Sep 26 03:44:30 dist-test-slave-n2th krb5kdc[6360](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394270, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:44:30.563647 779 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 26 03:44:30 dist-test-slave-n2th krb5kdc[6360](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394270, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:44:30.564885 779 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 26 03:44:30 dist-test-slave-n2th krb5kdc[6360](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394270, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.194.193@KRBTEST.COM
I20260926 03:44:30.578246 6868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:30.566138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48248 (local address 127.0.194.193:37847)
0926 03:44:30.566308 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0926 03:44:30.566314 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:44:30.566469 (+ 155us) server_negotiation.cc:407] Connection header received
0926 03:44:30.566577 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:44:30.566580 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:44:30.566628 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:44:30.566699 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:44:30.567645 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:30.568334 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:30.569465 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:30.569856 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:30.569940 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:30.573319 (+ 3379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:44:30.573324 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:44:30.573327 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:44:30.573357 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:44:30.575989 (+ 2632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:30.576597 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:30.576600 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:30.576603 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:30.576739 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:30.577075 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:30.577078 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:30.577080 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:30.577326 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:44:30.577540 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0926 03:44:30.577725 (+ 185us) server_negotiation.cc:299] Negotiation successful
0926 03:44:30.577918 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 03:44:30.579828 779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu pid 6466
I20260926 03:44:30.591475 6868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:30.580511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48254 (local address 127.0.194.193:37847)
0926 03:44:30.580699 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0926 03:44:30.580707 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:44:30.581045 (+ 338us) server_negotiation.cc:407] Connection header received
0926 03:44:30.581134 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:44:30.581138 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:44:30.581192 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:44:30.581278 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:44:30.582090 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:30.582875 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:30.584015 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:30.584439 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:30.584530 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:30.586370 (+ 1840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:44:30.586374 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:44:30.586377 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:44:30.586405 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:44:30.589060 (+ 2655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:30.589608 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:30.589611 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:30.589613 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:30.589738 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:30.590150 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:30.590153 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:30.590155 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:30.590395 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:44:30.590574 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0926 03:44:30.590961 (+ 387us) server_negotiation.cc:299] Negotiation successful
0926 03:44:30.591113 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 03:44:30.592099 6589 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:44:30.674162 779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu with pid 6466
I20260926 03:44:30.690346 779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu pid 6600
Sep 26 03:44:30 dist-test-slave-n2th krb5kdc[6360](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394270, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.194.194@KRBTEST.COM
I20260926 03:44:30.705669 6881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:30.691350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56158 (local address 127.0.194.194:35259)
0926 03:44:30.691839 (+ 489us) server_negotiation.cc:206] Beginning negotiation
0926 03:44:30.691846 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:44:30.691869 (+ 23us) server_negotiation.cc:407] Connection header received
0926 03:44:30.691925 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:44:30.691931 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:44:30.692133 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:44:30.692264 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:44:30.693187 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:30.694184 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:30.695437 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:30.695971 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:30.696066 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:30.699944 (+ 3878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:44:30.699955 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:44:30.699968 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:44:30.699997 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:44:30.702871 (+ 2874us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:30.703563 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:30.703573 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:30.703582 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:30.703698 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:30.704084 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:30.704091 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:30.704094 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:30.704423 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:44:30.704722 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0926 03:44:30.705042 (+ 320us) server_negotiation.cc:299] Negotiation successful
0926 03:44:30.705288 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":354,"thread_start_us":243,"threads_started":1}
I20260926 03:44:30.707051 6723 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:44:30.787640 779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu with pid 6600
I20260926 03:44:30.803442 779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu pid 6733
Sep 26 03:44:30 dist-test-slave-n2th krb5kdc[6360](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394270, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.194.195@KRBTEST.COM
I20260926 03:44:30.818502 6883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:30.804431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50374 (local address 127.0.194.195:38939)
0926 03:44:30.805052 (+ 621us) server_negotiation.cc:206] Beginning negotiation
0926 03:44:30.805059 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:44:30.805088 (+ 29us) server_negotiation.cc:407] Connection header received
0926 03:44:30.805147 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:44:30.805153 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:44:30.805386 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:44:30.805513 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:44:30.806663 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:30.807730 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:30.808924 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:30.809497 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:30.809595 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:30.813520 (+ 3925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:44:30.813530 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:44:30.813547 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:44:30.813580 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:44:30.816175 (+ 2595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:30.816766 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:30.816773 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:30.816775 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:30.816881 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:30.817186 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:30.817188 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:30.817190 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:30.817520 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:44:30.817733 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0926 03:44:30.817948 (+ 215us) server_negotiation.cc:299] Negotiation successful
0926 03:44:30.818179 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":462,"thread_start_us":309,"threads_started":1}
I20260926 03:44:30.819847 6856 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:44:30.911505 779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu with pid 6733
I20260926 03:44:30.927484 779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu pid 6373
I20260926 03:44:30.928684 6435 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:44:31.029794 779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu with pid 6373
2026-09-26T03:44:31Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (3921 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:44:31 dist-test-slave-n2th krb5kdc[6888](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:44:31 dist-test-slave-n2th krb5kdc[6888](info): set up 2 sockets
Sep 26 03:44:31 dist-test-slave-n2th krb5kdc[6888](info): commencing operation
krb5kdc: starting...
W20260926 03:44:32.127686 779 mini_kdc.cc:121] Time spent starting KDC: real 1.056s user 0.001s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/krb5kdc/test-user.keytab.
Sep 26 03:44:32 dist-test-slave-n2th krb5kdc[6888](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394272, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:44:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:44:32Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.194.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.194.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.194.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/krb5kdc/oryx_127.0.194.254.keytab.
Entry for principal oryx/127.0.194.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/krb5kdc/oryx_127.0.194.254.keytab.
I20260926 03:44:32.340161 779 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
/tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.194.254:44939
--webserver_interface=127.0.194.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/krb5kdc/oryx_127.0.194.254.keytab
--principal=oryx/127.0.194.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.194.212:35975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.194.254:44939
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:44:32.661659 6901 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:44:32.661955 6901 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:44:32.662112 6901 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:44:32.669348 6901 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:44:32.669457 6901 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:44:32.669538 6901 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:44:32.669620 6901 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:44:32.669715 6901 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:44:32.679786 6901 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.194.212:35975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.194.254:44939
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/krb5kdc/oryx_127.0.194.254.keytab
--principal=oryx/127.0.194.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.194.254:44939
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.194.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:44:32.681128 6901 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:44:32.682502 6901 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:44:32.694365 6909 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:44:32.694314 6907 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:44:32.695061 6906 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:44:32.695134 6901 server_base.cc:1034] running on GCE node
I20260926 03:44:32.695984 6901 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:44:32.697383 6901 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:44:32.698604 6901 hybrid_clock.cc:648] HybridClock initialized: now 1790394272698556 us; error 64 us; skew 500 ppm
I20260926 03:44:32.699012 6901 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:44:32 dist-test-slave-n2th krb5kdc[6888](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394272, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.194.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:44:32.702200 6901 init.cc:377] Logged in from keytab as oryx/127.0.194.254@KRBTEST.COM (short username oryx)
I20260926 03:44:32.703701 6901 webserver.cc:460] Webserver started at http://127.0.194.254:38891/ using document root <none> and password file <none>
I20260926 03:44:32.704427 6901 fs_manager.cc:359] Metadata directory not provided
I20260926 03:44:32.704573 6901 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:44:32.704879 6901 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:44:32.707942 6901 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/minicluster-data/master-0/data/instance:
uuid: "be6fefc6a90b41218affcf0df9459735"
format_stamp: "Formatted at 2026-09-26 03:44:32 on dist-test-slave-n2th"
server_key: "accf3e263d69af94d58449b0826f3471"
server_key_iv: "02336d73c72c6f84c9cc0dbf04d2c37b"
server_key_version: "encryptionkey@0"
I20260926 03:44:32.708711 6901 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/minicluster-data/master-0/wal/instance:
uuid: "be6fefc6a90b41218affcf0df9459735"
format_stamp: "Formatted at 2026-09-26 03:44:32 on dist-test-slave-n2th"
server_key: "accf3e263d69af94d58449b0826f3471"
server_key_iv: "02336d73c72c6f84c9cc0dbf04d2c37b"
server_key_version: "encryptionkey@0"
I20260926 03:44:32.714121 6901 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 03:44:32.717865 6916 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:44:32.719249 6901 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 03:44:32.719430 6901 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/minicluster-data/master-0/data,/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/minicluster-data/master-0/wal
uuid: "be6fefc6a90b41218affcf0df9459735"
format_stamp: "Formatted at 2026-09-26 03:44:32 on dist-test-slave-n2th"
server_key: "accf3e263d69af94d58449b0826f3471"
server_key_iv: "02336d73c72c6f84c9cc0dbf04d2c37b"
server_key_version: "encryptionkey@0"
I20260926 03:44:32.719633 6901 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:44:32.737023 6901 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:44:32.741452 6901 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:44:32.741748 6901 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:44:32.759896 6901 rpc_server.cc:307] RPC server started. Bound to: 127.0.194.254:44939
I20260926 03:44:32.759980 6967 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.194.254:44939 every 8 connection(s)
I20260926 03:44:32.761264 6901 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/minicluster-data/master-0/data/info.pb
I20260926 03:44:32.764869 6968 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:44:32.770349 779 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu as pid 6901
I20260926 03:44:32.770545 779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/minicluster-data/master-0/wal/instance
I20260926 03:44:32.771219 779 external_mini_cluster.cc:1428] Setting key 86e5140c174385beffae639aa8451e5b
I20260926 03:44:32.775563 6968 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be6fefc6a90b41218affcf0df9459735: Bootstrap starting.
I20260926 03:44:32.779353 6968 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P be6fefc6a90b41218affcf0df9459735: Neither blocks nor log segments found. Creating new log.
I20260926 03:44:32.780556 6968 log.cc:824] T 00000000000000000000000000000000 P be6fefc6a90b41218affcf0df9459735: Log is configured to *not* fsync() on all Append() calls
I20260926 03:44:32.783776 6968 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be6fefc6a90b41218affcf0df9459735: No bootstrap required, opened a new log
Sep 26 03:44:32 dist-test-slave-n2th krb5kdc[6888](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394272, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.194.254@KRBTEST.COM
I20260926 03:44:32.791800 6968 raft_consensus.cc:348] T 00000000000000000000000000000000 P be6fefc6a90b41218affcf0df9459735 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be6fefc6a90b41218affcf0df9459735" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 44939 } }
I20260926 03:44:32.792169 6968 raft_consensus.cc:374] T 00000000000000000000000000000000 P be6fefc6a90b41218affcf0df9459735 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:44:32.792289 6968 raft_consensus.cc:729] T 00000000000000000000000000000000 P be6fefc6a90b41218affcf0df9459735 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be6fefc6a90b41218affcf0df9459735, State: Initialized, Role: FOLLOWER
I20260926 03:44:32.792928 6968 consensus_queue.cc:260] T 00000000000000000000000000000000 P be6fefc6a90b41218affcf0df9459735 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be6fefc6a90b41218affcf0df9459735" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 44939 } }
I20260926 03:44:32.793222 6968 raft_consensus.cc:388] T 00000000000000000000000000000000 P be6fefc6a90b41218affcf0df9459735 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:44:32.793377 6968 raft_consensus.cc:482] T 00000000000000000000000000000000 P be6fefc6a90b41218affcf0df9459735 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:44:32.793555 6968 raft_consensus.cc:3037] T 00000000000000000000000000000000 P be6fefc6a90b41218affcf0df9459735 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:44:32.794381 6971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:32.773097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33042 (local address 127.0.194.254:44939)
0926 03:44:32.773946 (+ 849us) server_negotiation.cc:206] Beginning negotiation
0926 03:44:32.773966 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0926 03:44:32.774005 (+ 39us) server_negotiation.cc:407] Connection header received
0926 03:44:32.775655 (+ 1650us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:44:32.775698 (+ 43us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:44:32.776082 (+ 384us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:44:32.776383 (+ 301us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:44:32.778386 (+ 2003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:32.779823 (+ 1437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:32.781493 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:32.782308 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:32.782444 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:32.787105 (+ 4661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:44:32.787113 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:44:32.787131 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:44:32.787169 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:44:32.790217 (+ 3048us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:32.791000 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:32.791017 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:32.791024 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:32.791189 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:32.791565 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:32.791569 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:32.791571 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:32.792101 (+ 530us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:44:32.792451 (+ 350us) server_negotiation.cc:1035] Waiting for connection context
0926 03:44:32.792785 (+ 334us) server_negotiation.cc:299] Negotiation successful
0926 03:44:32.793249 (+ 464us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":572,"thread_start_us":377,"threads_started":1}
I20260926 03:44:32.795974 6968 raft_consensus.cc:504] T 00000000000000000000000000000000 P be6fefc6a90b41218affcf0df9459735 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be6fefc6a90b41218affcf0df9459735" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 44939 } }
I20260926 03:44:32.796458 6968 leader_election.cc:302] T 00000000000000000000000000000000 P be6fefc6a90b41218affcf0df9459735 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: be6fefc6a90b41218affcf0df9459735; no voters:
I20260926 03:44:32.797138 6968 leader_election.cc:288] T 00000000000000000000000000000000 P be6fefc6a90b41218affcf0df9459735 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:44:32.797250 6973 raft_consensus.cc:2781] T 00000000000000000000000000000000 P be6fefc6a90b41218affcf0df9459735 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:44:32.797904 6973 raft_consensus.cc:686] T 00000000000000000000000000000000 P be6fefc6a90b41218affcf0df9459735 [term 1 LEADER]: Becoming Leader. State: Replica: be6fefc6a90b41218affcf0df9459735, State: Running, Role: LEADER
I20260926 03:44:32.798456 6973 consensus_queue.cc:237] T 00000000000000000000000000000000 P be6fefc6a90b41218affcf0df9459735 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be6fefc6a90b41218affcf0df9459735" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 44939 } }
I20260926 03:44:32.799379 6968 sys_catalog.cc:564] T 00000000000000000000000000000000 P be6fefc6a90b41218affcf0df9459735 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:44:32.801132 6975 sys_catalog.cc:455] T 00000000000000000000000000000000 P be6fefc6a90b41218affcf0df9459735 [sys.catalog]: SysCatalogTable state changed. Reason: New leader be6fefc6a90b41218affcf0df9459735. Latest consensus state: current_term: 1 leader_uuid: "be6fefc6a90b41218affcf0df9459735" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be6fefc6a90b41218affcf0df9459735" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 44939 } } }
I20260926 03:44:32.801450 6975 sys_catalog.cc:458] T 00000000000000000000000000000000 P be6fefc6a90b41218affcf0df9459735 [sys.catalog]: This master's current role is: LEADER
I20260926 03:44:32.802966 6974 sys_catalog.cc:455] T 00000000000000000000000000000000 P be6fefc6a90b41218affcf0df9459735 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "be6fefc6a90b41218affcf0df9459735" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be6fefc6a90b41218affcf0df9459735" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 44939 } } }
I20260926 03:44:32.803329 6974 sys_catalog.cc:458] T 00000000000000000000000000000000 P be6fefc6a90b41218affcf0df9459735 [sys.catalog]: This master's current role is: LEADER
I20260926 03:44:32.807580 6978 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:44:32.813062 6978 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:44:32.824159 6978 catalog_manager.cc:1347] Generated new cluster ID: 5900f64ad7ae4b009c7ea641aa872bad
I20260926 03:44:32.824360 6978 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:44:32.850148 6978 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:44:32.851600 6978 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:44:32.859458 6978 catalog_manager.cc:5868] T 00000000000000000000000000000000 P be6fefc6a90b41218affcf0df9459735: Generated new TSK 0
I20260926 03:44:32.860419 6978 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.194.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.194.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.194.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/krb5kdc/oryx_127.0.194.193.keytab.
Entry for principal oryx/127.0.194.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/krb5kdc/oryx_127.0.194.193.keytab.
I20260926 03:44:32.920081 779 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
/tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.194.193:0
--local_ip_for_outbound_sockets=127.0.194.193
--webserver_interface=127.0.194.193
--webserver_port=0
--tserver_master_addrs=127.0.194.254:44939
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/krb5kdc/oryx_127.0.194.193.keytab
--principal=oryx/127.0.194.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.194.212:35975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:44:33.256524 6994 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:44:33.256829 6994 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:44:33.256996 6994 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:44:33.264585 6994 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:44:33.264719 6994 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:44:33.264914 6994 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.194.193
I20260926 03:44:33.275272 6994 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.194.212:35975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/krb5kdc/oryx_127.0.194.193.keytab
--principal=oryx/127.0.194.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.194.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.194.193
--webserver_port=0
--tserver_master_addrs=127.0.194.254:44939
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.194.193
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:44:33.276635 6994 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:44:33.278007 6994 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:44:33.291718 7000 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:44:33.291806 6999 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:44:33.291783 7002 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:44:33.292939 6994 server_base.cc:1034] running on GCE node
I20260926 03:44:33.293627 6994 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:44:33.294607 6994 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:44:33.295858 6994 hybrid_clock.cc:648] HybridClock initialized: now 1790394273295803 us; error 52 us; skew 500 ppm
I20260926 03:44:33.296203 6994 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:44:33 dist-test-slave-n2th krb5kdc[6888](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394273, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.194.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:44:33.299508 6994 init.cc:377] Logged in from keytab as oryx/127.0.194.193@KRBTEST.COM (short username oryx)
I20260926 03:44:33.301062 6994 webserver.cc:460] Webserver started at http://127.0.194.193:38631/ using document root <none> and password file <none>
I20260926 03:44:33.301781 6994 fs_manager.cc:359] Metadata directory not provided
I20260926 03:44:33.301925 6994 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:44:33.302248 6994 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:44:33.305299 6994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/minicluster-data/ts-0/data/instance:
uuid: "b08e95643f1a4377b8c2d42ac8762a70"
format_stamp: "Formatted at 2026-09-26 03:44:33 on dist-test-slave-n2th"
server_key: "8c4e532c88a083b3efed8ae4b2f9f88e"
server_key_iv: "76aaae7dc44e31c11b365444fb467be4"
server_key_version: "encryptionkey@0"
I20260926 03:44:33.306102 6994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/minicluster-data/ts-0/wal/instance:
uuid: "b08e95643f1a4377b8c2d42ac8762a70"
format_stamp: "Formatted at 2026-09-26 03:44:33 on dist-test-slave-n2th"
server_key: "8c4e532c88a083b3efed8ae4b2f9f88e"
server_key_iv: "76aaae7dc44e31c11b365444fb467be4"
server_key_version: "encryptionkey@0"
I20260926 03:44:33.311311 6994 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260926 03:44:33.314890 7009 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:44:33.316205 6994 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 03:44:33.316380 6994 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/minicluster-data/ts-0/wal
uuid: "b08e95643f1a4377b8c2d42ac8762a70"
format_stamp: "Formatted at 2026-09-26 03:44:33 on dist-test-slave-n2th"
server_key: "8c4e532c88a083b3efed8ae4b2f9f88e"
server_key_iv: "76aaae7dc44e31c11b365444fb467be4"
server_key_version: "encryptionkey@0"
I20260926 03:44:33.316588 6994 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:44:33.350531 6994 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:44:33.354667 6994 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:44:33.355013 6994 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:44:33.356016 6994 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:44:33.357445 6994 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:44:33.357553 6994 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:44:33.357712 6994 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:44:33.357782 6994 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:44:33.391274 6994 rpc_server.cc:307] RPC server started. Bound to: 127.0.194.193:32927
I20260926 03:44:33.391317 7121 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.194.193:32927 every 8 connection(s)
I20260926 03:44:33.392609 6994 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/minicluster-data/ts-0/data/info.pb
I20260926 03:44:33.402136 779 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu as pid 6994
I20260926 03:44:33.402379 779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/minicluster-data/ts-0/wal/instance
I20260926 03:44:33.403244 779 external_mini_cluster.cc:1428] Setting key a6647906a28aa999c5c7a0ce98d3d2a4
Sep 26 03:44:33 dist-test-slave-n2th krb5kdc[6888](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394273, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.194.193@KRBTEST.COM for oryx/127.0.194.254@KRBTEST.COM
I20260926 03:44:33.412601 7124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:33.395177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.193:44791 (local address 127.0.194.254:44939)
0926 03:44:33.395623 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0926 03:44:33.395627 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:44:33.396564 (+ 937us) server_negotiation.cc:407] Connection header received
0926 03:44:33.398324 (+ 1760us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:44:33.398329 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:44:33.398425 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:44:33.398542 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:44:33.400079 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:33.400828 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:33.402427 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:33.403107 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:33.403262 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:33.406802 (+ 3540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:44:33.406807 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:44:33.406810 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:44:33.406843 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:44:33.409413 (+ 2570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:33.410154 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:33.410160 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:33.410165 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:33.410281 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:33.410707 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:33.410713 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:33.410716 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:33.411013 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:44:33.411166 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0926 03:44:33.411955 (+ 789us) server_negotiation.cc:299] Negotiation successful
0926 03:44:33.412169 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351,"thread_start_us":251,"threads_started":1}
I20260926 03:44:33.413900 7125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:33.395719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:44939 (local address 127.0.194.193:44791)
0926 03:44:33.396411 (+ 692us) negotiation.cc:107] Waiting for socket to connect
0926 03:44:33.396449 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 03:44:33.398066 (+ 1617us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:44:33.398830 (+ 764us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:44:33.398845 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:44:33.399350 (+ 505us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:44:33.399913 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:33.399930 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:33.400992 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:33.400996 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:33.402230 (+ 1234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:33.402241 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:33.403339 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:33.403346 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:33.403585 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:33.404279 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:44:33.404316 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:44:33.406564 (+ 2248us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:44:33.409562 (+ 2998us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:44:33.409571 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:44:33.409589 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:44:33.410026 (+ 437us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:44:33.410397 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:44:33.410402 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:44:33.410406 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:44:33.410595 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:44:33.411137 (+ 542us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:44:33.411145 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:44:33.411575 (+ 430us) client_negotiation.cc:769] Sending connection context
0926 03:44:33.411899 (+ 324us) client_negotiation.cc:240] Negotiation successful
0926 03:44:33.412436 (+ 537us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":302,"threads_started":1}
I20260926 03:44:33.415513 7122 heartbeater.cc:347] Connected to a master server at 127.0.194.254:44939
I20260926 03:44:33.415819 7122 heartbeater.cc:464] Registering TS with master...
I20260926 03:44:33.416545 7122 heartbeater.cc:511] Master 127.0.194.254:44939 requested a full tablet report, sending...
I20260926 03:44:33.418653 6933 ts_manager.cc:194] Registered new tserver with Master: b08e95643f1a4377b8c2d42ac8762a70 (127.0.194.193:32927)
I20260926 03:44:33.420215 6933 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.194.193@KRBTEST.COM'} at 127.0.194.193:44791
WARNING: no policy specified for oryx/127.0.194.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.194.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.194.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/krb5kdc/oryx_127.0.194.194.keytab.
Entry for principal oryx/127.0.194.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/krb5kdc/oryx_127.0.194.194.keytab.
I20260926 03:44:33.451320 779 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
/tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.194.194:0
--local_ip_for_outbound_sockets=127.0.194.194
--webserver_interface=127.0.194.194
--webserver_port=0
--tserver_master_addrs=127.0.194.254:44939
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/krb5kdc/oryx_127.0.194.194.keytab
--principal=oryx/127.0.194.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.194.212:35975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:44:33.776502 7128 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:44:33.776811 7128 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:44:33.776980 7128 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:44:33.784322 7128 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:44:33.784471 7128 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:44:33.784650 7128 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.194.194
I20260926 03:44:33.794431 7128 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.194.212:35975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/krb5kdc/oryx_127.0.194.194.keytab
--principal=oryx/127.0.194.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.194.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.194.194
--webserver_port=0
--tserver_master_addrs=127.0.194.254:44939
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.194.194
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:44:33.795773 7128 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:44:33.797092 7128 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:44:33.810359 7134 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:44:33.810437 7136 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:44:33.810398 7133 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:44:33.811158 7128 server_base.cc:1034] running on GCE node
I20260926 03:44:33.811790 7128 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:44:33.812732 7128 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:44:33.813959 7128 hybrid_clock.cc:648] HybridClock initialized: now 1790394273813878 us; error 70 us; skew 500 ppm
I20260926 03:44:33.814296 7128 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:44:33 dist-test-slave-n2th krb5kdc[6888](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394273, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.194.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:44:33.817456 7128 init.cc:377] Logged in from keytab as oryx/127.0.194.194@KRBTEST.COM (short username oryx)
I20260926 03:44:33.818934 7128 webserver.cc:460] Webserver started at http://127.0.194.194:35031/ using document root <none> and password file <none>
I20260926 03:44:33.819589 7128 fs_manager.cc:359] Metadata directory not provided
I20260926 03:44:33.819689 7128 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:44:33.819984 7128 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:44:33.823218 7128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/minicluster-data/ts-1/data/instance:
uuid: "949d66f722dd4b5c86409272002b87b3"
format_stamp: "Formatted at 2026-09-26 03:44:33 on dist-test-slave-n2th"
server_key: "aaf9511636928b1f9d1b93be37657084"
server_key_iv: "0a09fee6cac4ecc693509f989c1062c2"
server_key_version: "encryptionkey@0"
I20260926 03:44:33.824005 7128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/minicluster-data/ts-1/wal/instance:
uuid: "949d66f722dd4b5c86409272002b87b3"
format_stamp: "Formatted at 2026-09-26 03:44:33 on dist-test-slave-n2th"
server_key: "aaf9511636928b1f9d1b93be37657084"
server_key_iv: "0a09fee6cac4ecc693509f989c1062c2"
server_key_version: "encryptionkey@0"
I20260926 03:44:33.829267 7128 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 03:44:33.832669 7143 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:44:33.833962 7128 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 03:44:33.834095 7128 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/minicluster-data/ts-1/wal
uuid: "949d66f722dd4b5c86409272002b87b3"
format_stamp: "Formatted at 2026-09-26 03:44:33 on dist-test-slave-n2th"
server_key: "aaf9511636928b1f9d1b93be37657084"
server_key_iv: "0a09fee6cac4ecc693509f989c1062c2"
server_key_version: "encryptionkey@0"
I20260926 03:44:33.834259 7128 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:44:33.851653 7128 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:44:33.856172 7128 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:44:33.856428 7128 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:44:33.857439 7128 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:44:33.858798 7128 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:44:33.858888 7128 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:44:33.858978 7128 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:44:33.859020 7128 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:44:33.892486 7128 rpc_server.cc:307] RPC server started. Bound to: 127.0.194.194:42589
I20260926 03:44:33.892529 7255 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.194.194:42589 every 8 connection(s)
I20260926 03:44:33.893857 7128 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/minicluster-data/ts-1/data/info.pb
I20260926 03:44:33.901631 779 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu as pid 7128
I20260926 03:44:33.901822 779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/minicluster-data/ts-1/wal/instance
I20260926 03:44:33.902459 779 external_mini_cluster.cc:1428] Setting key 80d37b3c1cb8a135b731b9941d4f5aae
Sep 26 03:44:33 dist-test-slave-n2th krb5kdc[6888](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394273, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.194.194@KRBTEST.COM for oryx/127.0.194.254@KRBTEST.COM
I20260926 03:44:33.917024 7124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:33.896657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.194:46357 (local address 127.0.194.254:44939)
0926 03:44:33.896839 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0926 03:44:33.896844 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:44:33.898128 (+ 1284us) server_negotiation.cc:407] Connection header received
0926 03:44:33.900181 (+ 2053us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:44:33.900187 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:44:33.900284 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:44:33.900421 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:44:33.902400 (+ 1979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:33.903242 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:33.905457 (+ 2215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:33.905893 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:33.906008 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:33.910416 (+ 4408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:44:33.910421 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:44:33.910424 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:44:33.910458 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:44:33.913435 (+ 2977us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:33.914478 (+ 1043us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:33.914482 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:33.914484 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:33.914626 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:33.915215 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:33.915219 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:33.915221 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:33.915504 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:44:33.915611 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0926 03:44:33.916581 (+ 970us) server_negotiation.cc:299] Negotiation successful
0926 03:44:33.916761 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 03:44:33.918674 7258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:33.897200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:44939 (local address 127.0.194.194:46357)
0926 03:44:33.897954 (+ 754us) negotiation.cc:107] Waiting for socket to connect
0926 03:44:33.897999 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 03:44:33.899896 (+ 1897us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:44:33.900677 (+ 781us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:44:33.900686 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:44:33.901263 (+ 577us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:44:33.902171 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:33.902199 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:33.903403 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:33.903407 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:33.905285 (+ 1878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:33.905301 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:33.906141 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:33.906151 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:33.906526 (+ 375us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:33.907305 (+ 779us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:44:33.907339 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:44:33.910126 (+ 2787us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:44:33.913589 (+ 3463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:44:33.913602 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:44:33.913626 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:44:33.914328 (+ 702us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:44:33.914811 (+ 483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:44:33.914819 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:44:33.914825 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:44:33.915094 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:44:33.915647 (+ 553us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:44:33.915659 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:44:33.916191 (+ 532us) client_negotiation.cc:769] Sending connection context
0926 03:44:33.916516 (+ 325us) client_negotiation.cc:240] Negotiation successful
0926 03:44:33.917125 (+ 609us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":278,"threads_started":1}
I20260926 03:44:33.920240 7256 heartbeater.cc:347] Connected to a master server at 127.0.194.254:44939
I20260926 03:44:33.920617 7256 heartbeater.cc:464] Registering TS with master...
I20260926 03:44:33.921370 7256 heartbeater.cc:511] Master 127.0.194.254:44939 requested a full tablet report, sending...
I20260926 03:44:33.923058 6933 ts_manager.cc:194] Registered new tserver with Master: 949d66f722dd4b5c86409272002b87b3 (127.0.194.194:42589)
I20260926 03:44:33.924314 6933 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.194.194@KRBTEST.COM'} at 127.0.194.194:46357
WARNING: no policy specified for oryx/127.0.194.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.194.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.194.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/krb5kdc/oryx_127.0.194.195.keytab.
Entry for principal oryx/127.0.194.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/krb5kdc/oryx_127.0.194.195.keytab.
I20260926 03:44:33.953703 779 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
/tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.194.195:0
--local_ip_for_outbound_sockets=127.0.194.195
--webserver_interface=127.0.194.195
--webserver_port=0
--tserver_master_addrs=127.0.194.254:44939
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/krb5kdc/oryx_127.0.194.195.keytab
--principal=oryx/127.0.194.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.194.212:35975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:44:34.276741 7261 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:44:34.277009 7261 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:44:34.277099 7261 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:44:34.284150 7261 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:44:34.284312 7261 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:44:34.284435 7261 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.194.195
I20260926 03:44:34.294050 7261 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.194.212:35975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/krb5kdc/oryx_127.0.194.195.keytab
--principal=oryx/127.0.194.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.194.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.194.195
--webserver_port=0
--tserver_master_addrs=127.0.194.254:44939
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.194.195
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:44:34.295297 7261 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:44:34.296566 7261 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:44:34.309689 7269 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:44:34.309674 7267 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:44:34.310015 7266 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:44:34.310838 7261 server_base.cc:1034] running on GCE node
I20260926 03:44:34.311506 7261 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:44:34.312582 7261 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:44:34.313809 7261 hybrid_clock.cc:648] HybridClock initialized: now 1790394274313757 us; error 47 us; skew 500 ppm
I20260926 03:44:34.314145 7261 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:44:34 dist-test-slave-n2th krb5kdc[6888](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394274, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.194.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:44:34.317423 7261 init.cc:377] Logged in from keytab as oryx/127.0.194.195@KRBTEST.COM (short username oryx)
I20260926 03:44:34.318868 7261 webserver.cc:460] Webserver started at http://127.0.194.195:32783/ using document root <none> and password file <none>
I20260926 03:44:34.319588 7261 fs_manager.cc:359] Metadata directory not provided
I20260926 03:44:34.319687 7261 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:44:34.319936 7261 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:44:34.322983 7261 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/minicluster-data/ts-2/data/instance:
uuid: "162b5f9a168f49db9ec3b1a5b0f0a526"
format_stamp: "Formatted at 2026-09-26 03:44:34 on dist-test-slave-n2th"
server_key: "ee8152e098a63fcbbadaa07bd1ffa5f1"
server_key_iv: "2984b4c524cc32929e9033fc7c332814"
server_key_version: "encryptionkey@0"
I20260926 03:44:34.323824 7261 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/minicluster-data/ts-2/wal/instance:
uuid: "162b5f9a168f49db9ec3b1a5b0f0a526"
format_stamp: "Formatted at 2026-09-26 03:44:34 on dist-test-slave-n2th"
server_key: "ee8152e098a63fcbbadaa07bd1ffa5f1"
server_key_iv: "2984b4c524cc32929e9033fc7c332814"
server_key_version: "encryptionkey@0"
I20260926 03:44:34.329006 7261 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260926 03:44:34.332594 7276 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:44:34.333956 7261 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 03:44:34.334124 7261 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/minicluster-data/ts-2/wal
uuid: "162b5f9a168f49db9ec3b1a5b0f0a526"
format_stamp: "Formatted at 2026-09-26 03:44:34 on dist-test-slave-n2th"
server_key: "ee8152e098a63fcbbadaa07bd1ffa5f1"
server_key_iv: "2984b4c524cc32929e9033fc7c332814"
server_key_version: "encryptionkey@0"
I20260926 03:44:34.334332 7261 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:44:34.352285 7261 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:44:34.356245 7261 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:44:34.356621 7261 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:44:34.357568 7261 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:44:34.358914 7261 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:44:34.358985 7261 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:44:34.359134 7261 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:44:34.359189 7261 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:44:34.391255 7261 rpc_server.cc:307] RPC server started. Bound to: 127.0.194.195:34367
I20260926 03:44:34.391382 7388 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.194.195:34367 every 8 connection(s)
I20260926 03:44:34.392573 7261 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/minicluster-data/ts-2/data/info.pb
I20260926 03:44:34.394155 779 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu as pid 7261
I20260926 03:44:34.394336 779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/minicluster-data/ts-2/wal/instance
I20260926 03:44:34.394958 779 external_mini_cluster.cc:1428] Setting key c4ab78cab28c15e190f08a51fbd58fdb
Sep 26 03:44:34 dist-test-slave-n2th krb5kdc[6888](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394274, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.194.195@KRBTEST.COM for oryx/127.0.194.254@KRBTEST.COM
I20260926 03:44:34.417439 7124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:34.395473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.195:55803 (local address 127.0.194.254:44939)
0926 03:44:34.395847 (+ 374us) server_negotiation.cc:206] Beginning negotiation
0926 03:44:34.395855 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:44:34.398164 (+ 2309us) server_negotiation.cc:407] Connection header received
0926 03:44:34.400054 (+ 1890us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:44:34.400058 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:44:34.400147 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:44:34.400229 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:44:34.401625 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:34.402346 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:34.403877 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:34.404267 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:34.404357 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:34.411669 (+ 7312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:44:34.411676 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:44:34.411681 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:44:34.411725 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:44:34.414278 (+ 2553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:34.415123 (+ 845us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:34.415129 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:34.415133 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:34.415279 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:34.415721 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:34.415726 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:34.415731 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:34.416002 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:44:34.416156 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0926 03:44:34.416864 (+ 708us) server_negotiation.cc:299] Negotiation successful
0926 03:44:34.417075 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":229}
I20260926 03:44:34.418380 7391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:34.396163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:44939 (local address 127.0.194.195:55803)
0926 03:44:34.398009 (+ 1846us) negotiation.cc:107] Waiting for socket to connect
0926 03:44:34.398065 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0926 03:44:34.399797 (+ 1732us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:44:34.400433 (+ 636us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:44:34.400442 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:44:34.400929 (+ 487us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:44:34.401486 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:34.401502 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:34.402474 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:34.402477 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:34.403754 (+ 1277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:34.403765 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:34.404449 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:34.404453 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:34.404641 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:34.408962 (+ 4321us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:44:34.408990 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:44:34.411459 (+ 2469us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:44:34.414434 (+ 2975us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:44:34.414444 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:44:34.414461 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:44:34.414994 (+ 533us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:44:34.415406 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:44:34.415411 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:44:34.415415 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:44:34.415602 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:44:34.416132 (+ 530us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:44:34.416141 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:44:34.416489 (+ 348us) client_negotiation.cc:769] Sending connection context
0926 03:44:34.416765 (+ 276us) client_negotiation.cc:240] Negotiation successful
0926 03:44:34.417297 (+ 532us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":812,"thread_start_us":381,"threads_started":1}
I20260926 03:44:34.419872 7389 heartbeater.cc:347] Connected to a master server at 127.0.194.254:44939
I20260926 03:44:34.420197 7389 heartbeater.cc:464] Registering TS with master...
I20260926 03:44:34.421106 7389 heartbeater.cc:511] Master 127.0.194.254:44939 requested a full tablet report, sending...
I20260926 03:44:34.422859 6932 ts_manager.cc:194] Registered new tserver with Master: 162b5f9a168f49db9ec3b1a5b0f0a526 (127.0.194.195:34367)
I20260926 03:44:34.423286 7122 heartbeater.cc:503] Master 127.0.194.254:44939 was elected leader, sending a full tablet report...
I20260926 03:44:34.423812 6932 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.194.195@KRBTEST.COM'} at 127.0.194.195:55803
I20260926 03:44:34.434963 779 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.194.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.194.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.194.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/krb5kdc/oryx_127.0.194.253.keytab.
Entry for principal oryx/127.0.194.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/krb5kdc/oryx_127.0.194.253.keytab.
I20260926 03:44:34.480170 779 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
/tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.194.253:37485
--webserver_interface=127.0.194.253
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/krb5kdc/oryx_127.0.194.253.keytab
--principal=oryx/127.0.194.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.194.212:35975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.194.254:44939,127.0.194.253:37485
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:44:34.812328 7394 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:44:34.812637 7394 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:44:34.812727 7394 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:44:34.820236 7394 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:44:34.820364 7394 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:44:34.820411 7394 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:44:34.820468 7394 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:44:34.820505 7394 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:44:34.831205 7394 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.194.212:35975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.194.254:44939,127.0.194.253:37485
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/krb5kdc/oryx_127.0.194.253.keytab
--principal=oryx/127.0.194.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.194.253:37485
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.194.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
W20260926 03:44:34.831553 7394 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.194.254:44939,127.0.194.253:37485'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260926 03:44:34.832855 7394 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:44:34.834115 7394 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:44:34.846285 7399 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:44:34.846335 7400 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:44:34.846333 7402 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:44:34.852519 7394 server_base.cc:1034] running on GCE node
I20260926 03:44:34.853636 7394 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:44:34.855610 7394 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:44:34.856866 7394 hybrid_clock.cc:648] HybridClock initialized: now 1790394274856814 us; error 52 us; skew 500 ppm
I20260926 03:44:34.857251 7394 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:44:34 dist-test-slave-n2th krb5kdc[6888](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394274, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.194.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:44:34.865617 7394 init.cc:377] Logged in from keytab as oryx/127.0.194.253@KRBTEST.COM (short username oryx)
I20260926 03:44:34.867400 7394 webserver.cc:460] Webserver started at http://127.0.194.253:34995/ using document root <none> and password file <none>
I20260926 03:44:34.868170 7394 fs_manager.cc:359] Metadata directory not provided
I20260926 03:44:34.868309 7394 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:44:34.868633 7394 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:44:34.871744 7394 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/minicluster-data/master-1/data/instance:
uuid: "af7004004c624669ae81014cf255d63f"
format_stamp: "Formatted at 2026-09-26 03:44:34 on dist-test-slave-n2th"
server_key: "bd324d199b27baeb450f0668836b730c"
server_key_iv: "3c5719251539893e0376a66b37bde9eb"
server_key_version: "encryptionkey@0"
I20260926 03:44:34.872529 7394 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/minicluster-data/master-1/wal/instance:
uuid: "af7004004c624669ae81014cf255d63f"
format_stamp: "Formatted at 2026-09-26 03:44:34 on dist-test-slave-n2th"
server_key: "bd324d199b27baeb450f0668836b730c"
server_key_iv: "3c5719251539893e0376a66b37bde9eb"
server_key_version: "encryptionkey@0"
I20260926 03:44:34.877610 7394 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 03:44:34.881103 7409 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:44:34.882437 7394 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260926 03:44:34.882640 7394 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/minicluster-data/master-1/data,/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/minicluster-data/master-1/wal
uuid: "af7004004c624669ae81014cf255d63f"
format_stamp: "Formatted at 2026-09-26 03:44:34 on dist-test-slave-n2th"
server_key: "bd324d199b27baeb450f0668836b730c"
server_key_iv: "3c5719251539893e0376a66b37bde9eb"
server_key_version: "encryptionkey@0"
I20260926 03:44:34.882884 7394 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:44:34.905702 7394 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:44:34.910101 7394 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:44:34.910379 7394 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:44:34.927158 7256 heartbeater.cc:503] Master 127.0.194.254:44939 was elected leader, sending a full tablet report...
I20260926 03:44:34.929812 7394 rpc_server.cc:307] RPC server started. Bound to: 127.0.194.253:37485
I20260926 03:44:34.929831 7460 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.194.253:37485 every 8 connection(s)
I20260926 03:44:34.931253 7394 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/minicluster-data/master-1/data/info.pb
I20260926 03:44:34.935079 7461 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:44:34.941025 779 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu as pid 7394
I20260926 03:44:34.942562 7461 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.194.254" port: 44939 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 03:44:34 dist-test-slave-n2th krb5kdc[6888](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394274, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.194.253@KRBTEST.COM for oryx/127.0.194.254@KRBTEST.COM
Sep 26 03:44:34 dist-test-slave-n2th krb5kdc[6888](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394272, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.194.253@KRBTEST.COM
I20260926 03:44:34.962280 7465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:34.945452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33048 (local address 127.0.194.254:44939)
0926 03:44:34.945990 (+ 538us) server_negotiation.cc:206] Beginning negotiation
0926 03:44:34.945996 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:44:34.946539 (+ 543us) server_negotiation.cc:407] Connection header received
0926 03:44:34.947058 (+ 519us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:44:34.947063 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:44:34.947164 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:44:34.947341 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:44:34.948690 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:34.949448 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:34.951246 (+ 1798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:34.951807 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:34.951903 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:34.955553 (+ 3650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:44:34.955559 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:44:34.955565 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:44:34.955605 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:44:34.958669 (+ 3064us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:34.959598 (+ 929us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:34.959601 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:34.959604 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:34.959723 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:34.960212 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:34.960217 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:34.960221 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:34.960640 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:44:34.960799 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0926 03:44:34.961664 (+ 865us) server_negotiation.cc:299] Negotiation successful
0926 03:44:34.961844 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":401,"thread_start_us":257,"threads_started":1}
I20260926 03:44:34.963512 7466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:34.945472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:44939 (local address 127.0.0.1:33048)
0926 03:44:34.946364 (+ 892us) negotiation.cc:107] Waiting for socket to connect
0926 03:44:34.946415 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 03:44:34.946786 (+ 371us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:44:34.947411 (+ 625us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:44:34.947418 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:44:34.948101 (+ 683us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:44:34.948511 (+ 410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:34.948529 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:34.949590 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:34.949595 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:34.951102 (+ 1507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:34.951113 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:34.952042 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:34.952049 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:34.952274 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:34.953060 (+ 786us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:44:34.953085 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:44:34.955378 (+ 2293us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:44:34.958841 (+ 3463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:44:34.958854 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:44:34.958870 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:44:34.959478 (+ 608us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:44:34.959832 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:44:34.959842 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:44:34.959846 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:44:34.960087 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:44:34.960784 (+ 697us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:44:34.960795 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:44:34.961221 (+ 426us) client_negotiation.cc:769] Sending connection context
0926 03:44:34.961531 (+ 310us) client_negotiation.cc:240] Negotiation successful
0926 03:44:34.962060 (+ 529us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":623,"thread_start_us":469,"threads_started":1}
I20260926 03:44:34.963533 7464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:34.943373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39670 (local address 127.0.194.253:37485)
0926 03:44:34.944267 (+ 894us) server_negotiation.cc:206] Beginning negotiation
0926 03:44:34.944279 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 03:44:34.944320 (+ 41us) server_negotiation.cc:407] Connection header received
0926 03:44:34.946645 (+ 2325us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:44:34.946671 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:44:34.947012 (+ 341us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:44:34.947191 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:44:34.948735 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:34.950156 (+ 1421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:34.951615 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:34.952266 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:34.952383 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:34.956913 (+ 4530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:44:34.956925 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:44:34.956942 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:44:34.956988 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:44:34.960045 (+ 3057us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:34.960834 (+ 789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:34.960842 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:34.960844 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:34.960983 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:34.961391 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:34.961396 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:34.961398 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:34.961788 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:44:34.962088 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0926 03:44:34.962922 (+ 834us) server_negotiation.cc:299] Negotiation successful
0926 03:44:34.963149 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":674,"thread_start_us":486,"threads_started":1}
I20260926 03:44:34.965626 7461 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.194.253" port: 37485 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 03:44:34 dist-test-slave-n2th krb5kdc[6888](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394274, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.194.253@KRBTEST.COM for oryx/127.0.194.253@KRBTEST.COM
I20260926 03:44:34.977193 7466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:34.966484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.253:37485 (local address 127.0.0.1:39680)
0926 03:44:34.966650 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0926 03:44:34.966666 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 03:44:34.966800 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:44:34.967083 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:44:34.967087 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:44:34.967406 (+ 319us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:44:34.967522 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:34.967530 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:34.968464 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:34.968470 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:34.969316 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:34.969326 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:34.969974 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:34.969979 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:34.970180 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:34.970673 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:44:34.970689 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:44:34.972608 (+ 1919us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:44:34.975226 (+ 2618us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:44:34.975229 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:44:34.975232 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:44:34.975594 (+ 362us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:44:34.975888 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:44:34.975891 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:44:34.975893 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:44:34.976003 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:44:34.976453 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:44:34.976456 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:44:34.976569 (+ 113us) client_negotiation.cc:769] Sending connection context
0926 03:44:34.976672 (+ 103us) client_negotiation.cc:240] Negotiation successful
0926 03:44:34.976909 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54,"mutex_wait_us":1}
I20260926 03:44:34.977234 7464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:34.966684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39680 (local address 127.0.194.253:37485)
0926 03:44:34.966853 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0926 03:44:34.966860 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:44:34.966876 (+ 16us) server_negotiation.cc:407] Connection header received
0926 03:44:34.966915 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:44:34.966919 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:44:34.966976 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:44:34.967070 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:44:34.967654 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:34.968331 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:34.969447 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:34.969833 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:34.969962 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:34.972775 (+ 2813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:44:34.972780 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:44:34.972783 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:44:34.972810 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:44:34.975094 (+ 2284us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:34.975690 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:34.975693 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:34.975695 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:34.975799 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:34.976091 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:34.976094 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:34.976095 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:34.976359 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:44:34.976469 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0926 03:44:34.976756 (+ 287us) server_negotiation.cc:299] Negotiation successful
0926 03:44:34.977014 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":139,"server-negotiator.queue_time_us":71}
I20260926 03:44:34.981806 7461 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af7004004c624669ae81014cf255d63f: Bootstrap starting.
I20260926 03:44:34.985065 7461 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P af7004004c624669ae81014cf255d63f: Neither blocks nor log segments found. Creating new log.
I20260926 03:44:34.986145 7461 log.cc:824] T 00000000000000000000000000000000 P af7004004c624669ae81014cf255d63f: Log is configured to *not* fsync() on all Append() calls
I20260926 03:44:34.989012 7461 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af7004004c624669ae81014cf255d63f: No bootstrap required, opened a new log
I20260926 03:44:34.996055 7461 raft_consensus.cc:348] T 00000000000000000000000000000000 P af7004004c624669ae81014cf255d63f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be6fefc6a90b41218affcf0df9459735" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 44939 } } peers { permanent_uuid: "af7004004c624669ae81014cf255d63f" member_type: VOTER last_known_addr { host: "127.0.194.253" port: 37485 } }
I20260926 03:44:34.996388 7461 raft_consensus.cc:374] T 00000000000000000000000000000000 P af7004004c624669ae81014cf255d63f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:44:34.996482 7461 raft_consensus.cc:729] T 00000000000000000000000000000000 P af7004004c624669ae81014cf255d63f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af7004004c624669ae81014cf255d63f, State: Initialized, Role: FOLLOWER
I20260926 03:44:34.997064 7461 consensus_queue.cc:260] T 00000000000000000000000000000000 P af7004004c624669ae81014cf255d63f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be6fefc6a90b41218affcf0df9459735" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 44939 } } peers { permanent_uuid: "af7004004c624669ae81014cf255d63f" member_type: VOTER last_known_addr { host: "127.0.194.253" port: 37485 } }
I20260926 03:44:34.997841 7468 sys_catalog.cc:455] T 00000000000000000000000000000000 P af7004004c624669ae81014cf255d63f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be6fefc6a90b41218affcf0df9459735" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 44939 } } peers { permanent_uuid: "af7004004c624669ae81014cf255d63f" member_type: VOTER last_known_addr { host: "127.0.194.253" port: 37485 } } }
I20260926 03:44:34.998203 7468 sys_catalog.cc:458] T 00000000000000000000000000000000 P af7004004c624669ae81014cf255d63f [sys.catalog]: This master's current role is: FOLLOWER
I20260926 03:44:34.999092 7461 sys_catalog.cc:564] T 00000000000000000000000000000000 P af7004004c624669ae81014cf255d63f [sys.catalog]: configured and running, proceeding with master startup.
W20260926 03:44:35.007771 7479 catalog_manager.cc:1558] T 00000000000000000000000000000000 P af7004004c624669ae81014cf255d63f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 03:44:35.007951 7479 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 03:44:35.010187 7394 master_runner.cc:428] Detected that this master af7004004c624669ae81014cf255d63f is joining an existing cluster
I20260926 03:44:35.010278 7394 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.194.253:37485
I20260926 03:44:35.043519 7464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:35.014240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39694 (local address 127.0.194.253:37485)
0926 03:44:35.014428 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0926 03:44:35.014433 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:44:35.016059 (+ 1626us) server_negotiation.cc:407] Connection header received
0926 03:44:35.016435 (+ 376us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:44:35.016442 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:44:35.016552 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:44:35.016676 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:44:35.019102 (+ 2426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:35.020209 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:35.022325 (+ 2116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:35.023402 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:35.023509 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:35.026552 (+ 3043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:44:35.026561 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:44:35.026569 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:44:35.026606 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:44:35.040431 (+ 13825us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:35.041236 (+ 805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:35.041240 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:35.041243 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:35.041393 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:35.041788 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:35.041793 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:35.041795 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:35.042183 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:44:35.042305 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0926 03:44:35.042861 (+ 556us) server_negotiation.cc:299] Negotiation successful
0926 03:44:35.043108 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 03:44:35.044602 7485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:35.014357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.253:37485 (local address 127.0.0.1:39694)
0926 03:44:35.015901 (+ 1544us) negotiation.cc:107] Waiting for socket to connect
0926 03:44:35.015948 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 03:44:35.016230 (+ 282us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:44:35.016966 (+ 736us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:44:35.016970 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:44:35.018285 (+ 1315us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:44:35.018957 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:35.018970 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:35.020469 (+ 1499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:35.020473 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:35.022136 (+ 1663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:35.022154 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:35.023872 (+ 1718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:35.023887 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:35.024306 (+ 419us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:35.024999 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:44:35.025026 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:44:35.026373 (+ 1347us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:44:35.040590 (+ 14217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:44:35.040595 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:44:35.040598 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:44:35.041113 (+ 515us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:44:35.041519 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:44:35.041523 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:44:35.041527 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:44:35.041677 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:44:35.042404 (+ 727us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:44:35.042408 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:44:35.042570 (+ 162us) client_negotiation.cc:769] Sending connection context
0926 03:44:35.043782 (+ 1212us) client_negotiation.cc:240] Negotiation successful
0926 03:44:35.044085 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1172,"mutex_wait_us":26,"thread_start_us":341,"threads_started":1}
WARNING: no policy specified for oryx/127.0.194.252@KRBTEST.COM; defaulting to no policy
I20260926 03:44:35.047986 7465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:35.014095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33054 (local address 127.0.194.254:44939)
0926 03:44:35.014309 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0926 03:44:35.014315 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:44:35.016496 (+ 2181us) server_negotiation.cc:407] Connection header received
0926 03:44:35.017108 (+ 612us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:44:35.017116 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:44:35.017218 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:44:35.017281 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:44:35.019119 (+ 1838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:35.020209 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:35.030127 (+ 9918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:35.030978 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:35.031142 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:35.041059 (+ 9917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:44:35.041066 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:44:35.041071 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:44:35.041112 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:44:35.044695 (+ 3583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:35.045491 (+ 796us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:35.045496 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:35.045499 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:35.045648 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:35.046119 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:35.046127 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:35.046130 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:35.046488 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:44:35.046609 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0926 03:44:35.047248 (+ 639us) server_negotiation.cc:299] Negotiation successful
0926 03:44:35.047556 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260926 03:44:35.049404 7486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:35.015196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:44939 (local address 127.0.0.1:33054)
0926 03:44:35.016417 (+ 1221us) negotiation.cc:107] Waiting for socket to connect
0926 03:44:35.016432 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 03:44:35.016955 (+ 523us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:44:35.017510 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:44:35.017516 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:44:35.018331 (+ 815us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:44:35.018963 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:35.018971 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:35.028446 (+ 9475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:35.028451 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:35.029973 (+ 1522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:35.029984 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:35.031425 (+ 1441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:35.031434 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:35.031644 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:35.038461 (+ 6817us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:44:35.038488 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:44:35.039821 (+ 1333us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:44:35.044855 (+ 5034us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:44:35.044859 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:44:35.044862 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:44:35.045338 (+ 476us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:44:35.045794 (+ 456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:44:35.045801 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:44:35.045805 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:44:35.045976 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:44:35.046699 (+ 723us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:44:35.046703 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:44:35.046953 (+ 250us) client_negotiation.cc:769] Sending connection context
0926 03:44:35.048243 (+ 1290us) client_negotiation.cc:240] Negotiation successful
0926 03:44:35.048638 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1069,"mutex_wait_us":44,"thread_start_us":7029,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.194.252@KRBTEST.COM" created.
Sep 26 03:44:35 dist-test-slave-n2th krb5kdc[6888](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394272, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.194.254@KRBTEST.COM for oryx/127.0.194.253@KRBTEST.COM
I20260926 03:44:35.075109 7464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:35.056521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39698 (local address 127.0.194.253:37485)
0926 03:44:35.056716 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0926 03:44:35.056725 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:44:35.057342 (+ 617us) server_negotiation.cc:407] Connection header received
0926 03:44:35.057694 (+ 352us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:44:35.057701 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:44:35.057795 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:44:35.057906 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:44:35.059622 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:35.060753 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:35.062925 (+ 2172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:35.063719 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:35.063838 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:35.068566 (+ 4728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:44:35.068575 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:44:35.068583 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:44:35.068628 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:44:35.072246 (+ 3618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:35.072990 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:35.072995 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:35.072999 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:35.073146 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:35.073527 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:35.073532 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:35.073535 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:35.073797 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:44:35.073979 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0926 03:44:35.074444 (+ 465us) server_negotiation.cc:299] Negotiation successful
0926 03:44:35.074677 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 03:44:35.075153 7488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:35.056487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.253:37485 (local address 127.0.0.1:39698)
0926 03:44:35.057220 (+ 733us) negotiation.cc:107] Waiting for socket to connect
0926 03:44:35.057254 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0926 03:44:35.057481 (+ 227us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:44:35.058166 (+ 685us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:44:35.058175 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:44:35.059097 (+ 922us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:44:35.059441 (+ 344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:35.059458 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:35.060918 (+ 1460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:35.060927 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:35.062709 (+ 1782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:35.062728 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:35.064114 (+ 1386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:35.064122 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:35.064385 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:35.065174 (+ 789us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:44:35.065201 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:44:35.068333 (+ 3132us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:44:35.072404 (+ 4071us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:44:35.072416 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:44:35.072425 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:44:35.072856 (+ 431us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:44:35.073257 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:44:35.073263 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:44:35.073267 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:44:35.073412 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:44:35.073979 (+ 567us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:44:35.073989 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:44:35.074159 (+ 170us) client_negotiation.cc:769] Sending connection context
0926 03:44:35.074338 (+ 179us) client_negotiation.cc:240] Negotiation successful
0926 03:44:35.074677 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":375,"threads_started":1}
I20260926 03:44:35.077454 6933 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.194.253:37485: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "af7004004c624669ae81014cf255d63f"
member_type: NON_VOTER
last_known_addr {
host: "127.0.194.253"
port: 37485
}
attrs {
promote: true
}
}
dest_uuid: "be6fefc6a90b41218affcf0df9459735"
cas_config_opid_index: -1
I20260926 03:44:35.078441 6933 consensus_queue.cc:237] T 00000000000000000000000000000000 P be6fefc6a90b41218affcf0df9459735 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "be6fefc6a90b41218affcf0df9459735" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 44939 } } peers { permanent_uuid: "af7004004c624669ae81014cf255d63f" member_type: NON_VOTER last_known_addr { host: "127.0.194.253" port: 37485 } attrs { promote: true } }
I20260926 03:44:35.082289 7492 raft_consensus.cc:2932] T 00000000000000000000000000000000 P be6fefc6a90b41218affcf0df9459735 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER af7004004c624669ae81014cf255d63f (127.0.194.253) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "be6fefc6a90b41218affcf0df9459735" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 44939 } } peers { permanent_uuid: "af7004004c624669ae81014cf255d63f" member_type: NON_VOTER last_known_addr { host: "127.0.194.253" port: 37485 } attrs { promote: true } } }
I20260926 03:44:35.083580 7436 raft_consensus.cc:3037] T 00000000000000000000000000000000 P af7004004c624669ae81014cf255d63f [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:44:35.086822 7492 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.194.253:37485
I20260926 03:44:35.087571 7394 master.cc:537] Master@127.0.194.253:37485 shutting down...
I20260926 03:44:35.087574 7491 sys_catalog.cc:455] T 00000000000000000000000000000000 P be6fefc6a90b41218affcf0df9459735 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "be6fefc6a90b41218affcf0df9459735" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "be6fefc6a90b41218affcf0df9459735" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 44939 } } peers { permanent_uuid: "af7004004c624669ae81014cf255d63f" member_type: NON_VOTER last_known_addr { host: "127.0.194.253" port: 37485 } attrs { promote: true } } }
I20260926 03:44:35.087540 7436 raft_consensus.cc:1261] T 00000000000000000000000000000000 P af7004004c624669ae81014cf255d63f [term 1 FOLLOWER]: Refusing update from remote peer be6fefc6a90b41218affcf0df9459735: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260926 03:44:35.087899 7491 sys_catalog.cc:458] T 00000000000000000000000000000000 P be6fefc6a90b41218affcf0df9459735 [sys.catalog]: This master's current role is: LEADER
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.194.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/krb5kdc/oryx_127.0.194.252.keytab.
Entry for principal oryx/127.0.194.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/krb5kdc/oryx_127.0.194.252.keytab.
I20260926 03:44:35.090050 779 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
/tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.194.252:34225
--webserver_interface=127.0.194.252
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/krb5kdc/oryx_127.0.194.252.keytab
--principal=oryx/127.0.194.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.194.212:35975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.194.254:44939,127.0.194.253:37485,127.0.194.252:34225
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 03:44:35.095854 7491 consensus_queue.cc:1035] T 00000000000000000000000000000000 P be6fefc6a90b41218affcf0df9459735 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af7004004c624669ae81014cf255d63f" member_type: NON_VOTER last_known_addr { host: "127.0.194.253" port: 37485 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 03:44:35.099331 6919 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260926 03:44:35.104326 6919 consensus_peers.cc:487] T 00000000000000000000000000000000 P be6fefc6a90b41218affcf0df9459735 -> Peer af7004004c624669ae81014cf255d63f (127.0.194.253:37485): Couldn't send request to peer af7004004c624669ae81014cf255d63f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260926 03:44:35.113029 7394 raft_consensus.cc:2229] T 00000000000000000000000000000000 P af7004004c624669ae81014cf255d63f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:44:35.113519 7394 raft_consensus.cc:2258] T 00000000000000000000000000000000 P af7004004c624669ae81014cf255d63f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:44:35.113698 7394 tablet_replica.cc:330] T 00000000000000000000000000000000 P af7004004c624669ae81014cf255d63f: stopping tablet replica
I20260926 03:44:35.120517 7394 master.cc:552] Master@127.0.194.253:37485 shutdown complete.
I20260926 03:44:35.120786 7394 master_runner.cc:305] Clearing existing system tablet
I20260926 03:44:35.121277 7394 fs_manager.cc:359] Metadata directory not provided
I20260926 03:44:35.121454 7394 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:44:35.124629 7394 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260926 03:44:35.126443 7497 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:44:35.126869 7394 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:44:35.127303 7394 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 03:44:35.127476 7394 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/minicluster-data/master-1/data,/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/minicluster-data/master-1/wal
uuid: "af7004004c624669ae81014cf255d63f"
format_stamp: "Formatted at 2026-09-26 03:44:34 on dist-test-slave-n2th"
server_key: "bd324d199b27baeb450f0668836b730c"
server_key_iv: "3c5719251539893e0376a66b37bde9eb"
server_key_version: "encryptionkey@0"
I20260926 03:44:35.132805 7394 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P af7004004c624669ae81014cf255d63f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:44:35.140365 7394 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P af7004004c624669ae81014cf255d63f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260926 03:44:35.140699 7394 log.cc:1196] T 00000000000000000000000000000000 P af7004004c624669ae81014cf255d63f: Deleting WAL directory at /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260926 03:44:35.141352 7394 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P af7004004c624669ae81014cf255d63f: Deleting consensus metadata
I20260926 03:44:35.141693 7394 master_runner.cc:315] Copying system tablet from 127.0.194.254:44939
I20260926 03:44:35.144326 7394 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P af7004004c624669ae81014cf255d63f: tablet copy: Beginning tablet copy session from remote peer at address 127.0.194.254:44939
I20260926 03:44:35.167047 7503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:35.145176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:44939 (local address 127.0.0.1:33066)
0926 03:44:35.151368 (+ 6192us) negotiation.cc:107] Waiting for socket to connect
0926 03:44:35.151397 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0926 03:44:35.151605 (+ 208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:44:35.152117 (+ 512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:44:35.152125 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:44:35.152695 (+ 570us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:44:35.153017 (+ 322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:35.153030 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:35.154511 (+ 1481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:35.154516 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:35.155982 (+ 1466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:35.155997 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:35.156998 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:35.157007 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:35.157232 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:35.158096 (+ 864us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:44:35.158127 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:44:35.159550 (+ 1423us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:44:35.163733 (+ 4183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:44:35.163740 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:44:35.163745 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:44:35.164287 (+ 542us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:44:35.164733 (+ 446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:44:35.164738 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:44:35.164741 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:44:35.164995 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:44:35.165761 (+ 766us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:44:35.165767 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:44:35.165952 (+ 185us) client_negotiation.cc:769] Sending connection context
0926 03:44:35.166104 (+ 152us) client_negotiation.cc:240] Negotiation successful
0926 03:44:35.166494 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6026,"thread_start_us":5764,"threads_started":1}
I20260926 03:44:35.167696 7465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:35.145270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33066 (local address 127.0.194.254:44939)
0926 03:44:35.145543 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0926 03:44:35.145548 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:44:35.151554 (+ 6006us) server_negotiation.cc:407] Connection header received
0926 03:44:35.151796 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:44:35.151803 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:44:35.151905 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:44:35.152030 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:44:35.153193 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:35.154336 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:35.156162 (+ 1826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:35.156809 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:35.156943 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:35.159807 (+ 2864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:44:35.159813 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:44:35.159819 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:44:35.159862 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:44:35.163564 (+ 3702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:35.164437 (+ 873us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:35.164443 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:35.164447 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:35.164606 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:35.165133 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:35.165139 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:35.165143 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:35.165546 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:44:35.165699 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0926 03:44:35.166770 (+ 1071us) server_negotiation.cc:299] Negotiation successful
0926 03:44:35.167025 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 03:44:35.168509 6953 tablet_copy_service.cc:140] P be6fefc6a90b41218affcf0df9459735: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer af7004004c624669ae81014cf255d63f ({username='oryx', principal='oryx/127.0.194.253@KRBTEST.COM'} at 127.0.0.1:33066)
I20260926 03:44:35.168835 6953 tablet_copy_service.cc:161] P be6fefc6a90b41218affcf0df9459735: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer af7004004c624669ae81014cf255d63f at {username='oryx', principal='oryx/127.0.194.253@KRBTEST.COM'} at 127.0.0.1:33066: session id = af7004004c624669ae81014cf255d63f-00000000000000000000000000000000
I20260926 03:44:35.173410 6953 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P be6fefc6a90b41218affcf0df9459735: Tablet Copy: opened 0 blocks and 1 log segments
I20260926 03:44:35.175737 7394 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260926 03:44:35.182832 7394 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P af7004004c624669ae81014cf255d63f: tablet copy: Starting download of 0 data blocks...
I20260926 03:44:35.183142 7394 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P af7004004c624669ae81014cf255d63f: tablet copy: Starting download of 1 WAL segments...
I20260926 03:44:35.185927 7394 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P af7004004c624669ae81014cf255d63f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260926 03:44:35.190326 6953 tablet_copy_service.cc:342] P be6fefc6a90b41218affcf0df9459735: Request end of tablet copy session af7004004c624669ae81014cf255d63f-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.194.253@KRBTEST.COM'} at 127.0.0.1:33066
I20260926 03:44:35.190558 6953 tablet_copy_service.cc:433] P be6fefc6a90b41218affcf0df9459735: ending tablet copy session af7004004c624669ae81014cf255d63f-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer af7004004c624669ae81014cf255d63f
I20260926 03:44:35.201709 7394 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:44:35.204419 7504 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:44:35.208496 7505 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:44:35.219101 7394 server_base.cc:1034] running on GCE node
W20260926 03:44:35.219491 7507 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:44:35.220069 7394 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:44:35.220894 7394 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:44:35.222153 7394 hybrid_clock.cc:648] HybridClock initialized: now 1790394275222096 us; error 39 us; skew 500 ppm
I20260926 03:44:35.222464 7394 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:44:35 dist-test-slave-n2th krb5kdc[6888](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394275, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.194.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:44:35.225119 7394 init.cc:377] Logged in from keytab as oryx/127.0.194.253@KRBTEST.COM (short username oryx)
I20260926 03:44:35.231674 7394 webserver.cc:460] Webserver started at http://127.0.194.253:35913/ using document root <none> and password file <none>
I20260926 03:44:35.232110 7394 fs_manager.cc:359] Metadata directory not provided
I20260926 03:44:35.232246 7394 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:44:35.235303 7394 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260926 03:44:35.237506 7514 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:44:35.237991 7394 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 03:44:35.238359 7394 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/minicluster-data/master-1/data,/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/minicluster-data/master-1/wal
uuid: "af7004004c624669ae81014cf255d63f"
format_stamp: "Formatted at 2026-09-26 03:44:34 on dist-test-slave-n2th"
server_key: "bd324d199b27baeb450f0668836b730c"
server_key_iv: "3c5719251539893e0376a66b37bde9eb"
server_key_version: "encryptionkey@0"
I20260926 03:44:35.238706 7394 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:44:35.248862 7394 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:44:35.252840 7394 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:44:35.276392 7394 rpc_server.cc:307] RPC server started. Bound to: 127.0.194.253:37485
I20260926 03:44:35.276429 7565 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.194.253:37485 every 8 connection(s)
I20260926 03:44:35.277493 7394 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/minicluster-data/master-1/data/info.pb
I20260926 03:44:35.282773 7566 sys_catalog.cc:263] Verifying existing consensus state
I20260926 03:44:35.286674 7566 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af7004004c624669ae81014cf255d63f: Bootstrap starting.
I20260926 03:44:35.302981 7566 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af7004004c624669ae81014cf255d63f: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 03:44:35.303490 7566 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af7004004c624669ae81014cf255d63f: Bootstrap complete.
I20260926 03:44:35.304523 7566 raft_consensus.cc:348] T 00000000000000000000000000000000 P af7004004c624669ae81014cf255d63f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "be6fefc6a90b41218affcf0df9459735" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 44939 } } peers { permanent_uuid: "af7004004c624669ae81014cf255d63f" member_type: NON_VOTER last_known_addr { host: "127.0.194.253" port: 37485 } attrs { promote: true } }
I20260926 03:44:35.304781 7566 raft_consensus.cc:729] T 00000000000000000000000000000000 P af7004004c624669ae81014cf255d63f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: af7004004c624669ae81014cf255d63f, State: Initialized, Role: LEARNER
I20260926 03:44:35.305027 7566 consensus_queue.cc:260] T 00000000000000000000000000000000 P af7004004c624669ae81014cf255d63f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "be6fefc6a90b41218affcf0df9459735" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 44939 } } peers { permanent_uuid: "af7004004c624669ae81014cf255d63f" member_type: NON_VOTER last_known_addr { host: "127.0.194.253" port: 37485 } attrs { promote: true } }
I20260926 03:44:35.306018 7569 sys_catalog.cc:455] T 00000000000000000000000000000000 P af7004004c624669ae81014cf255d63f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "be6fefc6a90b41218affcf0df9459735" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 44939 } } peers { permanent_uuid: "af7004004c624669ae81014cf255d63f" member_type: NON_VOTER last_known_addr { host: "127.0.194.253" port: 37485 } attrs { promote: true } } }
I20260926 03:44:35.306260 7566 sys_catalog.cc:564] T 00000000000000000000000000000000 P af7004004c624669ae81014cf255d63f [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:44:35.306412 7569 sys_catalog.cc:458] T 00000000000000000000000000000000 P af7004004c624669ae81014cf255d63f [sys.catalog]: This master's current role is: LEARNER
I20260926 03:44:35.316244 7580 catalog_manager.cc:1259] Loaded cluster ID: 5900f64ad7ae4b009c7ea641aa872bad
I20260926 03:44:35.316462 7580 catalog_manager.cc:1552] T 00000000000000000000000000000000 P af7004004c624669ae81014cf255d63f: loading cluster ID for follower catalog manager: success
I20260926 03:44:35.319974 7580 catalog_manager.cc:1574] T 00000000000000000000000000000000 P af7004004c624669ae81014cf255d63f: acquiring CA information for follower catalog manager: success
I20260926 03:44:35.322505 7580 catalog_manager.cc:1602] T 00000000000000000000000000000000 P af7004004c624669ae81014cf255d63f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 03:44:35.426353 7389 heartbeater.cc:503] Master 127.0.194.254:44939 was elected leader, sending a full tablet report...
W20260926 03:44:35.511333 7494 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:44:35.511713 7494 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:44:35.511868 7494 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:44:35.519241 7494 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:44:35.519399 7494 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:44:35.519500 7494 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:44:35.519601 7494 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:44:35.519676 7494 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:44:35.529671 7494 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.194.212:35975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.194.254:44939,127.0.194.253:37485,127.0.194.252:34225
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/krb5kdc/oryx_127.0.194.252.keytab
--principal=oryx/127.0.194.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.194.252:34225
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.194.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:44:35.531077 7494 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:44:35.532384 7494 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:44:35.544176 7585 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:44:35.544255 7586 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:44:35.544217 7588 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:44:35.545089 7494 server_base.cc:1034] running on GCE node
I20260926 03:44:35.545818 7494 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:44:35.547284 7494 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:44:35.548481 7494 hybrid_clock.cc:648] HybridClock initialized: now 1790394275548447 us; error 37 us; skew 500 ppm
I20260926 03:44:35.548807 7494 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:44:35 dist-test-slave-n2th krb5kdc[6888](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394275, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.194.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:44:35.551805 7494 init.cc:377] Logged in from keytab as oryx/127.0.194.252@KRBTEST.COM (short username oryx)
I20260926 03:44:35.553211 7494 webserver.cc:460] Webserver started at http://127.0.194.252:36277/ using document root <none> and password file <none>
I20260926 03:44:35.553835 7494 fs_manager.cc:359] Metadata directory not provided
I20260926 03:44:35.553933 7494 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:44:35.554171 7494 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:44:35.557041 7494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/minicluster-data/master-2/data/instance:
uuid: "037743be8b9845979457b84234b6bdad"
format_stamp: "Formatted at 2026-09-26 03:44:35 on dist-test-slave-n2th"
server_key: "44473f8b99c460498decc298506c1c17"
server_key_iv: "8189ba2a5953a63ae8b213b23a24bfa0"
server_key_version: "encryptionkey@0"
I20260926 03:44:35.557749 7494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/minicluster-data/master-2/wal/instance:
uuid: "037743be8b9845979457b84234b6bdad"
format_stamp: "Formatted at 2026-09-26 03:44:35 on dist-test-slave-n2th"
server_key: "44473f8b99c460498decc298506c1c17"
server_key_iv: "8189ba2a5953a63ae8b213b23a24bfa0"
server_key_version: "encryptionkey@0"
I20260926 03:44:35.562819 7494 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20260926 03:44:35.566108 7595 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:44:35.567489 7494 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 03:44:35.567659 7494 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/minicluster-data/master-2/data,/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/minicluster-data/master-2/wal
uuid: "037743be8b9845979457b84234b6bdad"
format_stamp: "Formatted at 2026-09-26 03:44:35 on dist-test-slave-n2th"
server_key: "44473f8b99c460498decc298506c1c17"
server_key_iv: "8189ba2a5953a63ae8b213b23a24bfa0"
server_key_version: "encryptionkey@0"
I20260926 03:44:35.567857 7494 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:44:35.584695 7494 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:44:35.588946 7494 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:44:35.589187 7494 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:44:35.607297 7494 rpc_server.cc:307] RPC server started. Bound to: 127.0.194.252:34225
I20260926 03:44:35.607365 7646 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.194.252:34225 every 8 connection(s)
I20260926 03:44:35.608741 7494 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/minicluster-data/master-2/data/info.pb
I20260926 03:44:35.609436 779 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu as pid 7494
I20260926 03:44:35.614428 7647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:44:35.621533 7647 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.194.254" port: 44939 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 03:44:35 dist-test-slave-n2th krb5kdc[6888](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394272, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.194.252@KRBTEST.COM
Sep 26 03:44:35 dist-test-slave-n2th krb5kdc[6888](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394275, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.194.252@KRBTEST.COM for oryx/127.0.194.254@KRBTEST.COM
I20260926 03:44:35.636298 7650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:35.611651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50430 (local address 127.0.194.252:34225)
0926 03:44:35.612404 (+ 753us) server_negotiation.cc:206] Beginning negotiation
0926 03:44:35.612418 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 03:44:35.612462 (+ 44us) server_negotiation.cc:407] Connection header received
0926 03:44:35.614144 (+ 1682us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:44:35.614168 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:44:35.614607 (+ 439us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:44:35.615070 (+ 463us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:44:35.616552 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:35.618076 (+ 1524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:35.619444 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:35.620171 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:35.620369 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:35.625134 (+ 4765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:44:35.625151 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:44:35.625172 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:44:35.625223 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:44:35.628946 (+ 3723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:35.629989 (+ 1043us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:35.630001 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:35.630014 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:35.630277 (+ 263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:35.630860 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:35.630867 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:35.630873 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:35.631712 (+ 839us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:44:35.633307 (+ 1595us) server_negotiation.cc:1035] Waiting for connection context
0926 03:44:35.633704 (+ 397us) server_negotiation.cc:299] Negotiation successful
0926 03:44:35.634474 (+ 770us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":536,"thread_start_us":289,"threads_started":1}
I20260926 03:44:35.644639 7651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:35.624265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:44939 (local address 127.0.0.1:33068)
0926 03:44:35.625087 (+ 822us) negotiation.cc:107] Waiting for socket to connect
0926 03:44:35.625137 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 03:44:35.625367 (+ 230us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:44:35.625732 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:44:35.625740 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:44:35.626638 (+ 898us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:44:35.627064 (+ 426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:35.627079 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:35.628221 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:35.628227 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:35.630177 (+ 1950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:35.630198 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:35.631862 (+ 1664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:35.631870 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:35.632194 (+ 324us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:35.633071 (+ 877us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:44:35.633110 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:44:35.637318 (+ 4208us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:44:35.641908 (+ 4590us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:44:35.641919 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:44:35.641926 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:44:35.642351 (+ 425us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:44:35.642801 (+ 450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:44:35.642806 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:44:35.642809 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:44:35.642956 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:44:35.643515 (+ 559us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:44:35.643523 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:44:35.643695 (+ 172us) client_negotiation.cc:769] Sending connection context
0926 03:44:35.643887 (+ 192us) client_negotiation.cc:240] Negotiation successful
0926 03:44:35.644211 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":543,"thread_start_us":404,"threads_started":1}
I20260926 03:44:35.644579 7465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:35.624371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33068 (local address 127.0.194.254:44939)
0926 03:44:35.624582 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0926 03:44:35.624590 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:44:35.625235 (+ 645us) server_negotiation.cc:407] Connection header received
0926 03:44:35.625522 (+ 287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:44:35.625526 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:44:35.625590 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:44:35.625662 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:44:35.627248 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:35.628067 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:35.630399 (+ 2332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:35.631338 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:35.631468 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:35.637632 (+ 6164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:44:35.637642 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:44:35.637651 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:44:35.637710 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:44:35.641759 (+ 4049us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:35.642484 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:35.642490 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:35.642494 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:35.642660 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:35.643070 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:35.643076 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:35.643080 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:35.643382 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:44:35.643535 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0926 03:44:35.643981 (+ 446us) server_negotiation.cc:299] Negotiation successful
0926 03:44:35.644191 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":59}
I20260926 03:44:35.645570 7654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:35.629364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39706 (local address 127.0.194.253:37485)
0926 03:44:35.638780 (+ 9416us) server_negotiation.cc:206] Beginning negotiation
0926 03:44:35.638788 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:44:35.639290 (+ 502us) server_negotiation.cc:407] Connection header received
0926 03:44:35.639416 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:44:35.639421 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:44:35.639517 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:44:35.639682 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:44:35.640908 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:35.642091 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:35.643764 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:35.644547 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:35.644696 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:35.644916 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0926 03:44:35.645100 (+ 184us) server_negotiation.cc:299] Negotiation successful
0926 03:44:35.645245 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9155,"thread_start_us":9014,"threads_started":1}
I20260926 03:44:35.645628 7653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:35.629213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.253:37485 (local address 127.0.0.1:39706)
0926 03:44:35.639182 (+ 9969us) negotiation.cc:107] Waiting for socket to connect
0926 03:44:35.639200 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 03:44:35.639306 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:44:35.640436 (+ 1130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:44:35.640441 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:44:35.640470 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:44:35.640759 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:35.640770 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:35.642220 (+ 1450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:35.642226 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:35.643607 (+ 1381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:35.643619 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:35.644718 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:35.644724 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:35.644898 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:35.644963 (+ 65us) client_negotiation.cc:769] Sending connection context
0926 03:44:35.645064 (+ 101us) client_negotiation.cc:240] Negotiation successful
0926 03:44:35.645195 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9842,"thread_start_us":9715,"threads_started":1}
I20260926 03:44:35.646436 7647 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.194.253" port: 37485 } has no permanent_uuid. Determining permanent_uuid...
I20260926 03:44:35.646601 7541 raft_consensus.cc:1203] T 00000000000000000000000000000000 P af7004004c624669ae81014cf255d63f [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5] Dedup: 1.5->[]
I20260926 03:44:35.647481 7569 sys_catalog.cc:455] T 00000000000000000000000000000000 P af7004004c624669ae81014cf255d63f [sys.catalog]: SysCatalogTable state changed. Reason: New leader be6fefc6a90b41218affcf0df9459735. Latest consensus state: current_term: 1 leader_uuid: "be6fefc6a90b41218affcf0df9459735" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "be6fefc6a90b41218affcf0df9459735" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 44939 } } peers { permanent_uuid: "af7004004c624669ae81014cf255d63f" member_type: NON_VOTER last_known_addr { host: "127.0.194.253" port: 37485 } attrs { promote: true } } }
I20260926 03:44:35.647737 7569 sys_catalog.cc:458] T 00000000000000000000000000000000 P af7004004c624669ae81014cf255d63f [sys.catalog]: This master's current role is: LEARNER
Sep 26 03:44:35 dist-test-slave-n2th krb5kdc[6888](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394275, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.194.252@KRBTEST.COM for oryx/127.0.194.253@KRBTEST.COM
I20260926 03:44:35.658495 7651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:35.647156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.253:37485 (local address 127.0.0.1:39720)
0926 03:44:35.647530 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0926 03:44:35.647549 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 03:44:35.647639 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:44:35.648108 (+ 469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:44:35.648112 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:44:35.648451 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:44:35.648578 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:35.648585 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:35.649452 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:35.649455 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:35.650348 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:35.650357 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:35.651076 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:35.651081 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:35.651204 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:35.651713 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:44:35.651729 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:44:35.653665 (+ 1936us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:44:35.656511 (+ 2846us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:44:35.656514 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:44:35.656517 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:44:35.656881 (+ 364us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:44:35.657247 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:44:35.657250 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:44:35.657252 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:44:35.657371 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:44:35.657811 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:44:35.657814 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:44:35.657932 (+ 118us) client_negotiation.cc:769] Sending connection context
0926 03:44:35.658039 (+ 107us) client_negotiation.cc:240] Negotiation successful
0926 03:44:35.658258 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68}
I20260926 03:44:35.658540 7654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:35.647385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39720 (local address 127.0.194.253:37485)
0926 03:44:35.647871 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0926 03:44:35.647883 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 03:44:35.647897 (+ 14us) server_negotiation.cc:407] Connection header received
0926 03:44:35.647933 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:44:35.647936 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:44:35.647989 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:44:35.648061 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:44:35.648687 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:35.649340 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:35.650470 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:35.650927 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:35.651016 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:35.653847 (+ 2831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:44:35.653852 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:44:35.653855 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:44:35.653886 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:44:35.656372 (+ 2486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:35.657002 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:35.657006 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:35.657008 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:35.657146 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:35.657457 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:35.657460 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:35.657462 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:35.657718 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:44:35.657823 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 03:44:35.658120 (+ 297us) server_negotiation.cc:299] Negotiation successful
0926 03:44:35.658288 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378}
I20260926 03:44:35.659607 7647 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.194.252" port: 34225 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 03:44:35 dist-test-slave-n2th krb5kdc[6888](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394275, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.194.252@KRBTEST.COM for oryx/127.0.194.252@KRBTEST.COM
I20260926 03:44:35.671912 7651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:35.660448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.252:34225 (local address 127.0.0.1:50432)
0926 03:44:35.660639 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0926 03:44:35.660659 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 03:44:35.660788 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:44:35.661065 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:44:35.661070 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:44:35.661422 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:44:35.661560 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:35.661567 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:35.662506 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:35.662512 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:35.663413 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:35.663424 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:35.664109 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:35.664115 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:35.664294 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:35.664873 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:44:35.664890 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:44:35.666874 (+ 1984us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:44:35.669485 (+ 2611us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:44:35.669490 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:44:35.669494 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:44:35.669833 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:44:35.670206 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:44:35.670213 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:44:35.670219 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:44:35.670362 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:44:35.670890 (+ 528us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:44:35.670895 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:44:35.671049 (+ 154us) client_negotiation.cc:769] Sending connection context
0926 03:44:35.671186 (+ 137us) client_negotiation.cc:240] Negotiation successful
0926 03:44:35.671456 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61}
I20260926 03:44:35.672003 7650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:35.660443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50432 (local address 127.0.194.252:34225)
0926 03:44:35.660633 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0926 03:44:35.660639 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:44:35.660728 (+ 89us) server_negotiation.cc:407] Connection header received
0926 03:44:35.660903 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:44:35.660906 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:44:35.660954 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:44:35.661053 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:44:35.661687 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:35.662379 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:35.663550 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:35.663948 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:35.664093 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:35.667048 (+ 2955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:44:35.667052 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:44:35.667056 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:44:35.667082 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:44:35.669346 (+ 2264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:35.669978 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:35.669984 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:35.669989 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:35.670103 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:35.670463 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:35.670471 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:35.670474 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:35.670784 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:44:35.670926 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0926 03:44:35.671253 (+ 327us) server_negotiation.cc:299] Negotiation successful
0926 03:44:35.671624 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":162,"server-negotiator.queue_time_us":63}
I20260926 03:44:35.676788 7647 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 037743be8b9845979457b84234b6bdad: Bootstrap starting.
I20260926 03:44:35.679838 7647 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 037743be8b9845979457b84234b6bdad: Neither blocks nor log segments found. Creating new log.
I20260926 03:44:35.680784 7647 log.cc:824] T 00000000000000000000000000000000 P 037743be8b9845979457b84234b6bdad: Log is configured to *not* fsync() on all Append() calls
I20260926 03:44:35.683306 7647 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 037743be8b9845979457b84234b6bdad: No bootstrap required, opened a new log
I20260926 03:44:35.690256 7647 raft_consensus.cc:348] T 00000000000000000000000000000000 P 037743be8b9845979457b84234b6bdad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be6fefc6a90b41218affcf0df9459735" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 44939 } } peers { permanent_uuid: "af7004004c624669ae81014cf255d63f" member_type: VOTER last_known_addr { host: "127.0.194.253" port: 37485 } } peers { permanent_uuid: "037743be8b9845979457b84234b6bdad" member_type: VOTER last_known_addr { host: "127.0.194.252" port: 34225 } }
I20260926 03:44:35.690618 7647 raft_consensus.cc:374] T 00000000000000000000000000000000 P 037743be8b9845979457b84234b6bdad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:44:35.690714 7647 raft_consensus.cc:729] T 00000000000000000000000000000000 P 037743be8b9845979457b84234b6bdad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 037743be8b9845979457b84234b6bdad, State: Initialized, Role: FOLLOWER
I20260926 03:44:35.691443 7647 consensus_queue.cc:260] T 00000000000000000000000000000000 P 037743be8b9845979457b84234b6bdad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be6fefc6a90b41218affcf0df9459735" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 44939 } } peers { permanent_uuid: "af7004004c624669ae81014cf255d63f" member_type: VOTER last_known_addr { host: "127.0.194.253" port: 37485 } } peers { permanent_uuid: "037743be8b9845979457b84234b6bdad" member_type: VOTER last_known_addr { host: "127.0.194.252" port: 34225 } }
I20260926 03:44:35.692488 7656 sys_catalog.cc:455] T 00000000000000000000000000000000 P 037743be8b9845979457b84234b6bdad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be6fefc6a90b41218affcf0df9459735" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 44939 } } peers { permanent_uuid: "af7004004c624669ae81014cf255d63f" member_type: VOTER last_known_addr { host: "127.0.194.253" port: 37485 } } peers { permanent_uuid: "037743be8b9845979457b84234b6bdad" member_type: VOTER last_known_addr { host: "127.0.194.252" port: 34225 } } }
I20260926 03:44:35.692759 7656 sys_catalog.cc:458] T 00000000000000000000000000000000 P 037743be8b9845979457b84234b6bdad [sys.catalog]: This master's current role is: FOLLOWER
I20260926 03:44:35.693526 7647 sys_catalog.cc:564] T 00000000000000000000000000000000 P 037743be8b9845979457b84234b6bdad [sys.catalog]: configured and running, proceeding with master startup.
W20260926 03:44:35.702988 7667 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 037743be8b9845979457b84234b6bdad: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 03:44:35.703153 7667 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 03:44:35.707056 7494 master_runner.cc:428] Detected that this master 037743be8b9845979457b84234b6bdad is joining an existing cluster
I20260926 03:44:35.707175 7494 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.194.252:34225
I20260926 03:44:35.737191 7650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:35.711379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50444 (local address 127.0.194.252:34225)
0926 03:44:35.711572 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0926 03:44:35.711580 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:44:35.719414 (+ 7834us) server_negotiation.cc:407] Connection header received
0926 03:44:35.719719 (+ 305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:44:35.719724 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:44:35.719790 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:44:35.719861 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:44:35.721277 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:35.722619 (+ 1342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:35.725455 (+ 2836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:35.726498 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:35.726626 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:35.729617 (+ 2991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:44:35.729623 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:44:35.729627 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:44:35.729671 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:44:35.733736 (+ 4065us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:35.734692 (+ 956us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:35.734697 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:35.734700 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:35.734887 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:35.735294 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:35.735300 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:35.735305 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:35.735800 (+ 495us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:44:35.735947 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0926 03:44:35.736508 (+ 561us) server_negotiation.cc:299] Negotiation successful
0926 03:44:35.736747 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":70,"server-negotiator.queue_time_us":59}
I20260926 03:44:35.738438 7673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:35.711419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.252:34225 (local address 127.0.0.1:50444)
0926 03:44:35.719285 (+ 7866us) negotiation.cc:107] Waiting for socket to connect
0926 03:44:35.719309 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 03:44:35.719557 (+ 248us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:44:35.720066 (+ 509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:44:35.720071 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:44:35.720729 (+ 658us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:44:35.721100 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:35.721117 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:35.722922 (+ 1805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:35.722930 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:35.725274 (+ 2344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:35.725294 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:35.727020 (+ 1726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:35.727026 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:35.727271 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:35.728186 (+ 915us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:44:35.728212 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:44:35.729391 (+ 1179us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:44:35.733887 (+ 4496us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:44:35.733891 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:44:35.733894 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:44:35.734554 (+ 660us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:44:35.735013 (+ 459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:44:35.735017 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:44:35.735020 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:44:35.735175 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:44:35.736026 (+ 851us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:44:35.736031 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:44:35.736235 (+ 204us) client_negotiation.cc:769] Sending connection context
0926 03:44:35.737586 (+ 1351us) client_negotiation.cc:240] Negotiation successful
0926 03:44:35.737905 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7689,"thread_start_us":459,"threads_started":1}
I20260926 03:44:35.740146 7654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:35.711712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39722 (local address 127.0.194.253:37485)
0926 03:44:35.711904 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0926 03:44:35.711911 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:44:35.722804 (+ 10893us) server_negotiation.cc:407] Connection header received
0926 03:44:35.723104 (+ 300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:44:35.723110 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:44:35.723197 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:44:35.723274 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:44:35.724868 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:35.725942 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:35.728905 (+ 2963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:35.729572 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:35.729677 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:35.732344 (+ 2667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:44:35.732350 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:44:35.732355 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:44:35.732393 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:44:35.735808 (+ 3415us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:35.736714 (+ 906us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:35.736720 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:35.736723 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:35.736875 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:35.737708 (+ 833us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:35.737713 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:35.737716 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:35.738098 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:44:35.738235 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0926 03:44:35.739502 (+ 1267us) server_negotiation.cc:299] Negotiation successful
0926 03:44:35.739752 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 03:44:35.741377 7675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:35.712413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.253:37485 (local address 127.0.0.1:39722)
0926 03:44:35.722644 (+ 10231us) negotiation.cc:107] Waiting for socket to connect
0926 03:44:35.722669 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0926 03:44:35.722928 (+ 259us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:44:35.723487 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:44:35.723492 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:44:35.724318 (+ 826us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:44:35.724667 (+ 349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:35.724681 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:35.726157 (+ 1476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:35.726162 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:35.727990 (+ 1828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:35.728006 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:35.729986 (+ 1980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:35.729996 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:35.730259 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:35.730830 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:44:35.730859 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:44:35.732105 (+ 1246us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:44:35.735984 (+ 3879us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:44:35.735989 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:44:35.735993 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:44:35.736561 (+ 568us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:44:35.737047 (+ 486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:44:35.737051 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:44:35.737055 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:44:35.737235 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:44:35.738337 (+ 1102us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:44:35.738343 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:44:35.738561 (+ 218us) client_negotiation.cc:769] Sending connection context
0926 03:44:35.740575 (+ 2014us) client_negotiation.cc:240] Negotiation successful
0926 03:44:35.740920 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10008,"mutex_wait_us":1,"thread_start_us":15725,"threads_started":1}
I20260926 03:44:35.744016 779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu pid 6994
I20260926 03:44:35.745473 7465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:35.711316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33070 (local address 127.0.194.254:44939)
0926 03:44:35.711511 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0926 03:44:35.711519 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:44:35.728701 (+ 17182us) server_negotiation.cc:407] Connection header received
0926 03:44:35.728864 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:44:35.728873 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:44:35.728956 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:44:35.729068 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:44:35.730490 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:35.731744 (+ 1254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:35.733892 (+ 2148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:35.734606 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:35.734701 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:35.737422 (+ 2721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:44:35.737429 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:44:35.737434 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:44:35.737478 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:44:35.741920 (+ 4442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:35.743157 (+ 1237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:35.743162 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:35.743166 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:35.743333 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:35.743776 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:35.743781 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:35.743784 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:35.744064 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:44:35.744224 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0926 03:44:35.744852 (+ 628us) server_negotiation.cc:299] Negotiation successful
0926 03:44:35.745101 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260926 03:44:35.745458 7674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:35.712477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:44939 (local address 127.0.0.1:33070)
0926 03:44:35.728561 (+ 16084us) negotiation.cc:107] Waiting for socket to connect
0926 03:44:35.728581 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 03:44:35.728726 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:44:35.729241 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:44:35.729246 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:44:35.729904 (+ 658us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:44:35.730298 (+ 394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:35.730313 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:35.731948 (+ 1635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:35.731955 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:35.733727 (+ 1772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:35.733736 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:35.735004 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:35.735010 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:35.735190 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:35.735797 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:44:35.735817 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:44:35.737177 (+ 1360us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:44:35.742241 (+ 5064us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:44:35.742245 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:44:35.742249 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:44:35.743014 (+ 765us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:44:35.743482 (+ 468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:44:35.743488 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:44:35.743492 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:44:35.743669 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:44:35.744308 (+ 639us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:44:35.744312 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:44:35.744497 (+ 185us) client_negotiation.cc:769] Sending connection context
0926 03:44:35.744677 (+ 180us) client_negotiation.cc:240] Negotiation successful
0926 03:44:35.744986 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15936,"thread_start_us":9367,"threads_started":1}
Sep 26 03:44:35 dist-test-slave-n2th krb5kdc[6888](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394272, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.194.193@KRBTEST.COM
Sep 26 03:44:35 dist-test-slave-n2th krb5kdc[6888](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394272, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.194.254@KRBTEST.COM for oryx/127.0.194.252@KRBTEST.COM
I20260926 03:44:35.765241 7653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:35.750956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.252:34225 (local address 127.0.0.1:50460)
0926 03:44:35.751205 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0926 03:44:35.751220 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 03:44:35.751342 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:44:35.751737 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:44:35.751744 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:44:35.752131 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:44:35.752272 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:35.752279 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:35.753492 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:35.753501 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:35.754367 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:35.754377 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:35.755194 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:35.755200 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:35.755380 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:35.755929 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:44:35.755945 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:44:35.758178 (+ 2233us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:44:35.762223 (+ 4045us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:44:35.762229 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:44:35.762233 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:44:35.762837 (+ 604us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:44:35.763269 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:44:35.763274 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:44:35.763278 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:44:35.763464 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:44:35.764206 (+ 742us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:44:35.764211 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:44:35.764401 (+ 190us) client_negotiation.cc:769] Sending connection context
0926 03:44:35.764567 (+ 166us) client_negotiation.cc:240] Negotiation successful
0926 03:44:35.764857 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145}
I20260926 03:44:35.766580 7650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:35.750933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50460 (local address 127.0.194.252:34225)
0926 03:44:35.751089 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0926 03:44:35.751095 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:44:35.751292 (+ 197us) server_negotiation.cc:407] Connection header received
0926 03:44:35.751522 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:44:35.751526 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:44:35.751603 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:44:35.751703 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:44:35.752416 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:35.753360 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:35.754510 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:35.755030 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:35.755168 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:35.758358 (+ 3190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:44:35.758363 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:44:35.758366 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:44:35.758400 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:44:35.762044 (+ 3644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:35.762981 (+ 937us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:35.762988 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:35.762994 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:35.763134 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:35.763611 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:35.763617 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:35.763622 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:35.764005 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:44:35.764145 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0926 03:44:35.765979 (+ 1834us) server_negotiation.cc:299] Negotiation successful
0926 03:44:35.766216 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 03:44:35.766677 7676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:35.745856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33658 (local address 127.0.194.193:32927)
0926 03:44:35.746508 (+ 652us) server_negotiation.cc:206] Beginning negotiation
0926 03:44:35.746519 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 03:44:35.746549 (+ 30us) server_negotiation.cc:407] Connection header received
0926 03:44:35.746625 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:44:35.746633 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:44:35.746952 (+ 319us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:44:35.747128 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:44:35.748549 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:35.749849 (+ 1300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:35.751823 (+ 1974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:35.752723 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:35.752859 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:35.758507 (+ 5648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:44:35.758523 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:44:35.758542 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:44:35.758583 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:44:35.762496 (+ 3913us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:35.763433 (+ 937us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:35.763443 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:35.763446 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:35.763596 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:35.764048 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:35.764057 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:35.764061 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:35.764490 (+ 429us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:44:35.764761 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0926 03:44:35.766028 (+ 1267us) server_negotiation.cc:299] Negotiation successful
0926 03:44:35.766283 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":480,"thread_start_us":635,"threads_started":1}
I20260926 03:44:35.768208 7117 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:44:35.768666 6933 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.194.252:34225: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "037743be8b9845979457b84234b6bdad"
member_type: NON_VOTER
last_known_addr {
host: "127.0.194.252"
port: 34225
}
attrs {
promote: true
}
}
dest_uuid: "be6fefc6a90b41218affcf0df9459735"
cas_config_opid_index: 5
I20260926 03:44:35.770013 6933 consensus_queue.cc:237] T 00000000000000000000000000000000 P be6fefc6a90b41218affcf0df9459735 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "be6fefc6a90b41218affcf0df9459735" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 44939 } } peers { permanent_uuid: "af7004004c624669ae81014cf255d63f" member_type: NON_VOTER last_known_addr { host: "127.0.194.253" port: 37485 } attrs { promote: true } } peers { permanent_uuid: "037743be8b9845979457b84234b6bdad" member_type: NON_VOTER last_known_addr { host: "127.0.194.252" port: 34225 } attrs { promote: true } }
I20260926 03:44:35.772398 7652 raft_consensus.cc:2932] T 00000000000000000000000000000000 P be6fefc6a90b41218affcf0df9459735 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 037743be8b9845979457b84234b6bdad (127.0.194.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "be6fefc6a90b41218affcf0df9459735" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 44939 } } peers { permanent_uuid: "af7004004c624669ae81014cf255d63f" member_type: NON_VOTER last_known_addr { host: "127.0.194.253" port: 37485 } attrs { promote: true } } peers { permanent_uuid: "037743be8b9845979457b84234b6bdad" member_type: NON_VOTER last_known_addr { host: "127.0.194.252" port: 34225 } attrs { promote: true } } }
I20260926 03:44:35.773442 7622 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 037743be8b9845979457b84234b6bdad [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:44:35.776554 7652 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.194.252:34225
I20260926 03:44:35.787808 7679 sys_catalog.cc:455] T 00000000000000000000000000000000 P be6fefc6a90b41218affcf0df9459735 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "be6fefc6a90b41218affcf0df9459735" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "be6fefc6a90b41218affcf0df9459735" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 44939 } } peers { permanent_uuid: "af7004004c624669ae81014cf255d63f" member_type: NON_VOTER last_known_addr { host: "127.0.194.253" port: 37485 } attrs { promote: true } } peers { permanent_uuid: "037743be8b9845979457b84234b6bdad" member_type: NON_VOTER last_known_addr { host: "127.0.194.252" port: 34225 } attrs { promote: true } } }
I20260926 03:44:35.788638 7679 sys_catalog.cc:458] T 00000000000000000000000000000000 P be6fefc6a90b41218affcf0df9459735 [sys.catalog]: This master's current role is: LEADER
I20260926 03:44:35.790242 7541 raft_consensus.cc:1261] T 00000000000000000000000000000000 P af7004004c624669ae81014cf255d63f [term 1 LEARNER]: Refusing update from remote peer be6fefc6a90b41218affcf0df9459735: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260926 03:44:35.790971 7680 consensus_queue.cc:1035] T 00000000000000000000000000000000 P be6fefc6a90b41218affcf0df9459735 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af7004004c624669ae81014cf255d63f" member_type: NON_VOTER last_known_addr { host: "127.0.194.253" port: 37485 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260926 03:44:35.793509 7541 raft_consensus.cc:2932] T 00000000000000000000000000000000 P af7004004c624669ae81014cf255d63f [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 037743be8b9845979457b84234b6bdad (127.0.194.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "be6fefc6a90b41218affcf0df9459735" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 44939 } } peers { permanent_uuid: "af7004004c624669ae81014cf255d63f" member_type: NON_VOTER last_known_addr { host: "127.0.194.253" port: 37485 } attrs { promote: true } } peers { permanent_uuid: "037743be8b9845979457b84234b6bdad" member_type: NON_VOTER last_known_addr { host: "127.0.194.252" port: 34225 } attrs { promote: true } } }
I20260926 03:44:35.795092 7494 master.cc:537] Master@127.0.194.252:34225 shutting down...
I20260926 03:44:35.795969 7622 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 037743be8b9845979457b84234b6bdad [term 1 FOLLOWER]: Refusing update from remote peer be6fefc6a90b41218affcf0df9459735: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260926 03:44:35.797042 7680 consensus_queue.cc:1035] T 00000000000000000000000000000000 P be6fefc6a90b41218affcf0df9459735 [LEADER]: Connected to new peer: Peer: permanent_uuid: "037743be8b9845979457b84234b6bdad" member_type: NON_VOTER last_known_addr { host: "127.0.194.252" port: 34225 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 03:44:35.812269 6918 consensus_peers.cc:487] T 00000000000000000000000000000000 P be6fefc6a90b41218affcf0df9459735 -> Peer 037743be8b9845979457b84234b6bdad (127.0.194.252:34225): Couldn't send request to peer 037743be8b9845979457b84234b6bdad. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260926 03:44:35.812963 7569 sys_catalog.cc:455] T 00000000000000000000000000000000 P af7004004c624669ae81014cf255d63f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "be6fefc6a90b41218affcf0df9459735" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "be6fefc6a90b41218affcf0df9459735" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 44939 } } peers { permanent_uuid: "af7004004c624669ae81014cf255d63f" member_type: NON_VOTER last_known_addr { host: "127.0.194.253" port: 37485 } attrs { promote: true } } peers { permanent_uuid: "037743be8b9845979457b84234b6bdad" member_type: NON_VOTER last_known_addr { host: "127.0.194.252" port: 34225 } attrs { promote: true } } }
I20260926 03:44:35.813302 7569 sys_catalog.cc:458] T 00000000000000000000000000000000 P af7004004c624669ae81014cf255d63f [sys.catalog]: This master's current role is: LEARNER
I20260926 03:44:35.814188 7494 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 037743be8b9845979457b84234b6bdad [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:44:35.814558 7679 raft_consensus.cc:1050] T 00000000000000000000000000000000 P be6fefc6a90b41218affcf0df9459735: attempting to promote NON_VOTER af7004004c624669ae81014cf255d63f to VOTER
I20260926 03:44:35.815514 7679 consensus_queue.cc:237] T 00000000000000000000000000000000 P be6fefc6a90b41218affcf0df9459735 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "be6fefc6a90b41218affcf0df9459735" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 44939 } } peers { permanent_uuid: "af7004004c624669ae81014cf255d63f" member_type: VOTER last_known_addr { host: "127.0.194.253" port: 37485 } attrs { promote: false } } peers { permanent_uuid: "037743be8b9845979457b84234b6bdad" member_type: NON_VOTER last_known_addr { host: "127.0.194.252" port: 34225 } attrs { promote: true } }
I20260926 03:44:35.816465 7494 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 037743be8b9845979457b84234b6bdad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:44:35.816704 7494 tablet_replica.cc:330] T 00000000000000000000000000000000 P 037743be8b9845979457b84234b6bdad: stopping tablet replica
I20260926 03:44:35.817647 7541 raft_consensus.cc:1261] T 00000000000000000000000000000000 P af7004004c624669ae81014cf255d63f [term 1 LEARNER]: Refusing update from remote peer be6fefc6a90b41218affcf0df9459735: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260926 03:44:35.818207 7678 consensus_queue.cc:1035] T 00000000000000000000000000000000 P be6fefc6a90b41218affcf0df9459735 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af7004004c624669ae81014cf255d63f" member_type: VOTER last_known_addr { host: "127.0.194.253" port: 37485 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20260926 03:44:35.820363 6918 consensus_peers.cc:487] T 00000000000000000000000000000000 P be6fefc6a90b41218affcf0df9459735 -> Peer 037743be8b9845979457b84234b6bdad (127.0.194.252:34225): Couldn't send request to peer 037743be8b9845979457b84234b6bdad. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260926 03:44:35.820881 7680 raft_consensus.cc:2932] T 00000000000000000000000000000000 P be6fefc6a90b41218affcf0df9459735 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, af7004004c624669ae81014cf255d63f (127.0.194.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "be6fefc6a90b41218affcf0df9459735" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 44939 } } peers { permanent_uuid: "af7004004c624669ae81014cf255d63f" member_type: VOTER last_known_addr { host: "127.0.194.253" port: 37485 } attrs { promote: false } } peers { permanent_uuid: "037743be8b9845979457b84234b6bdad" member_type: NON_VOTER last_known_addr { host: "127.0.194.252" port: 34225 } attrs { promote: true } } }
I20260926 03:44:35.825453 7494 master.cc:552] Master@127.0.194.252:34225 shutdown complete.
I20260926 03:44:35.825536 7494 master_runner.cc:305] Clearing existing system tablet
I20260926 03:44:35.825903 7494 fs_manager.cc:359] Metadata directory not provided
I20260926 03:44:35.826004 7494 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:44:35.825894 7652 sys_catalog.cc:455] T 00000000000000000000000000000000 P be6fefc6a90b41218affcf0df9459735 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "be6fefc6a90b41218affcf0df9459735" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "be6fefc6a90b41218affcf0df9459735" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 44939 } } peers { permanent_uuid: "af7004004c624669ae81014cf255d63f" member_type: VOTER last_known_addr { host: "127.0.194.253" port: 37485 } attrs { promote: false } } peers { permanent_uuid: "037743be8b9845979457b84234b6bdad" member_type: NON_VOTER last_known_addr { host: "127.0.194.252" port: 34225 } attrs { promote: true } } }
I20260926 03:44:35.826128 7652 sys_catalog.cc:458] T 00000000000000000000000000000000 P be6fefc6a90b41218affcf0df9459735 [sys.catalog]: This master's current role is: LEADER
I20260926 03:44:35.827317 7540 raft_consensus.cc:2932] T 00000000000000000000000000000000 P af7004004c624669ae81014cf255d63f [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, af7004004c624669ae81014cf255d63f (127.0.194.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "be6fefc6a90b41218affcf0df9459735" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 44939 } } peers { permanent_uuid: "af7004004c624669ae81014cf255d63f" member_type: VOTER last_known_addr { host: "127.0.194.253" port: 37485 } attrs { promote: false } } peers { permanent_uuid: "037743be8b9845979457b84234b6bdad" member_type: NON_VOTER last_known_addr { host: "127.0.194.252" port: 34225 } attrs { promote: true } } }
I20260926 03:44:35.831005 7652 sys_catalog.cc:455] T 00000000000000000000000000000000 P be6fefc6a90b41218affcf0df9459735 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "be6fefc6a90b41218affcf0df9459735" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "be6fefc6a90b41218affcf0df9459735" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 44939 } } peers { permanent_uuid: "af7004004c624669ae81014cf255d63f" member_type: VOTER last_known_addr { host: "127.0.194.253" port: 37485 } attrs { promote: false } } peers { permanent_uuid: "037743be8b9845979457b84234b6bdad" member_type: NON_VOTER last_known_addr { host: "127.0.194.252" port: 34225 } attrs { promote: true } } }
I20260926 03:44:35.831269 7652 sys_catalog.cc:458] T 00000000000000000000000000000000 P be6fefc6a90b41218affcf0df9459735 [sys.catalog]: This master's current role is: LEADER
I20260926 03:44:35.831965 7569 sys_catalog.cc:455] T 00000000000000000000000000000000 P af7004004c624669ae81014cf255d63f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "be6fefc6a90b41218affcf0df9459735" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "be6fefc6a90b41218affcf0df9459735" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 44939 } } peers { permanent_uuid: "af7004004c624669ae81014cf255d63f" member_type: VOTER last_known_addr { host: "127.0.194.253" port: 37485 } attrs { promote: false } } peers { permanent_uuid: "037743be8b9845979457b84234b6bdad" member_type: NON_VOTER last_known_addr { host: "127.0.194.252" port: 34225 } attrs { promote: true } } }
I20260926 03:44:35.832322 7569 sys_catalog.cc:458] T 00000000000000000000000000000000 P af7004004c624669ae81014cf255d63f [sys.catalog]: This master's current role is: FOLLOWER
I20260926 03:44:35.834801 7494 fs_manager.cc:709] Time spent opening directory manager: real 0.008s user 0.007s sys 0.001s
I20260926 03:44:35.838076 7685 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:44:35.838490 7494 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:44:35.838831 7494 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260926 03:44:35.838977 7494 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/minicluster-data/master-2/data,/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/minicluster-data/master-2/wal
uuid: "037743be8b9845979457b84234b6bdad"
format_stamp: "Formatted at 2026-09-26 03:44:35 on dist-test-slave-n2th"
server_key: "44473f8b99c460498decc298506c1c17"
server_key_iv: "8189ba2a5953a63ae8b213b23a24bfa0"
server_key_version: "encryptionkey@0"
I20260926 03:44:35.844280 7494 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 037743be8b9845979457b84234b6bdad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:44:35.851224 7494 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 037743be8b9845979457b84234b6bdad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260926 03:44:35.851393 7494 log.cc:1196] T 00000000000000000000000000000000 P 037743be8b9845979457b84234b6bdad: Deleting WAL directory at /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260926 03:44:35.851864 7494 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 037743be8b9845979457b84234b6bdad: Deleting consensus metadata
I20260926 03:44:35.852109 7494 master_runner.cc:315] Copying system tablet from 127.0.194.254:44939
I20260926 03:44:35.854633 7494 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 037743be8b9845979457b84234b6bdad: tablet copy: Beginning tablet copy session from remote peer at address 127.0.194.254:44939
I20260926 03:44:35.873493 7465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:35.858412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33074 (local address 127.0.194.254:44939)
0926 03:44:35.858646 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0926 03:44:35.858651 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:44:35.859337 (+ 686us) server_negotiation.cc:407] Connection header received
0926 03:44:35.859513 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:44:35.859519 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:44:35.859604 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:44:35.859722 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:44:35.860744 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:35.861848 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:35.863760 (+ 1912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:35.864443 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:35.864583 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:35.866948 (+ 2365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:44:35.866955 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:44:35.866961 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:44:35.867006 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:44:35.870540 (+ 3534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:35.871343 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:35.871350 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:35.871354 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:35.871507 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:35.871929 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:35.871934 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:35.871937 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:35.872294 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:44:35.872441 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0926 03:44:35.872903 (+ 462us) server_negotiation.cc:299] Negotiation successful
0926 03:44:35.873140 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260926 03:44:35.873531 7691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:35.856897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:44939 (local address 127.0.0.1:33074)
0926 03:44:35.859184 (+ 2287us) negotiation.cc:107] Waiting for socket to connect
0926 03:44:35.859209 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0926 03:44:35.859375 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:44:35.859777 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:44:35.859782 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:44:35.860323 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:44:35.860582 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:35.860593 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:35.862018 (+ 1425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:35.862024 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:35.863596 (+ 1572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:35.863610 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:35.864615 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:35.864621 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:35.864807 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:35.865563 (+ 756us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:44:35.865586 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:44:35.866660 (+ 1074us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:44:35.870694 (+ 4034us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:44:35.870700 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:44:35.870704 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:44:35.871201 (+ 497us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:44:35.871616 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:44:35.871621 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:44:35.871625 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:44:35.871806 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:44:35.872423 (+ 617us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:44:35.872428 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:44:35.872609 (+ 181us) client_negotiation.cc:769] Sending connection context
0926 03:44:35.872768 (+ 159us) client_negotiation.cc:240] Negotiation successful
0926 03:44:35.873101 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2155,"thread_start_us":1950,"threads_started":1}
I20260926 03:44:35.874490 6953 tablet_copy_service.cc:140] P be6fefc6a90b41218affcf0df9459735: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 037743be8b9845979457b84234b6bdad ({username='oryx', principal='oryx/127.0.194.252@KRBTEST.COM'} at 127.0.0.1:33074)
I20260926 03:44:35.874660 6953 tablet_copy_service.cc:161] P be6fefc6a90b41218affcf0df9459735: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 037743be8b9845979457b84234b6bdad at {username='oryx', principal='oryx/127.0.194.252@KRBTEST.COM'} at 127.0.0.1:33074: session id = 037743be8b9845979457b84234b6bdad-00000000000000000000000000000000
I20260926 03:44:35.878304 6953 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P be6fefc6a90b41218affcf0df9459735: Tablet Copy: opened 0 blocks and 1 log segments
I20260926 03:44:35.880136 7494 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260926 03:44:35.886678 7494 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 037743be8b9845979457b84234b6bdad: tablet copy: Starting download of 0 data blocks...
I20260926 03:44:35.887034 7494 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 037743be8b9845979457b84234b6bdad: tablet copy: Starting download of 1 WAL segments...
I20260926 03:44:35.889544 7494 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 037743be8b9845979457b84234b6bdad: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260926 03:44:35.893893 6953 tablet_copy_service.cc:342] P be6fefc6a90b41218affcf0df9459735: Request end of tablet copy session 037743be8b9845979457b84234b6bdad-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.194.252@KRBTEST.COM'} at 127.0.0.1:33074
I20260926 03:44:35.894029 6953 tablet_copy_service.cc:433] P be6fefc6a90b41218affcf0df9459735: ending tablet copy session 037743be8b9845979457b84234b6bdad-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 037743be8b9845979457b84234b6bdad
I20260926 03:44:35.906517 7494 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:44:35.909210 7692 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:44:35.912308 7693 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:44:35.917003 779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu with pid 6994
W20260926 03:44:35.923558 7695 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:44:35.923638 7494 server_base.cc:1034] running on GCE node
I20260926 03:44:35.924114 7494 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:44:35.924831 7494 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:44:35.926035 7494 hybrid_clock.cc:648] HybridClock initialized: now 1790394275925981 us; error 46 us; skew 500 ppm
I20260926 03:44:35.926293 7494 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:44:35 dist-test-slave-n2th krb5kdc[6888](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394275, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.194.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:44:35.928421 7494 init.cc:377] Logged in from keytab as oryx/127.0.194.252@KRBTEST.COM (short username oryx)
I20260926 03:44:35.932228 7494 webserver.cc:460] Webserver started at http://127.0.194.252:42805/ using document root <none> and password file <none>
I20260926 03:44:35.932549 7494 fs_manager.cc:359] Metadata directory not provided
I20260926 03:44:35.932662 7494 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:44:35.935103 7494 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260926 03:44:35.936722 7702 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:44:35.937179 7494 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20260926 03:44:35.937306 7494 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/minicluster-data/master-2/data,/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/minicluster-data/master-2/wal
uuid: "037743be8b9845979457b84234b6bdad"
format_stamp: "Formatted at 2026-09-26 03:44:35 on dist-test-slave-n2th"
server_key: "44473f8b99c460498decc298506c1c17"
server_key_iv: "8189ba2a5953a63ae8b213b23a24bfa0"
server_key_version: "encryptionkey@0"
I20260926 03:44:35.937554 7494 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:44:35.942960 779 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
/tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.194.193:32927
--local_ip_for_outbound_sockets=127.0.194.193
--tserver_master_addrs=127.0.194.254:44939
--webserver_port=38631
--webserver_interface=127.0.194.193
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/krb5kdc/oryx_127.0.194.193.keytab
--principal=oryx/127.0.194.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.194.212:35975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.194.254:44939,127.0.194.253:37485
--tserver_master_addrs=127.0.194.254:44939,127.0.194.253:37485,127.0.194.252:34225 with env {KRB5CCNAME=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 03:44:35.964501 7494 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:44:35.967958 7494 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:44:35.987528 7494 rpc_server.cc:307] RPC server started. Bound to: 127.0.194.252:34225
I20260926 03:44:35.987555 7754 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.194.252:34225 every 8 connection(s)
I20260926 03:44:35.988492 7494 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/minicluster-data/master-2/data/info.pb
I20260926 03:44:35.995674 7755 sys_catalog.cc:263] Verifying existing consensus state
I20260926 03:44:36.000151 7755 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 037743be8b9845979457b84234b6bdad: Bootstrap starting.
I20260926 03:44:36.015693 7755 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 037743be8b9845979457b84234b6bdad: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 03:44:36.016182 7755 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 037743be8b9845979457b84234b6bdad: Bootstrap complete.
I20260926 03:44:36.017036 7755 raft_consensus.cc:348] T 00000000000000000000000000000000 P 037743be8b9845979457b84234b6bdad [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "be6fefc6a90b41218affcf0df9459735" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 44939 } } peers { permanent_uuid: "af7004004c624669ae81014cf255d63f" member_type: VOTER last_known_addr { host: "127.0.194.253" port: 37485 } attrs { promote: false } } peers { permanent_uuid: "037743be8b9845979457b84234b6bdad" member_type: NON_VOTER last_known_addr { host: "127.0.194.252" port: 34225 } attrs { promote: true } }
I20260926 03:44:36.017306 7755 raft_consensus.cc:729] T 00000000000000000000000000000000 P 037743be8b9845979457b84234b6bdad [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 037743be8b9845979457b84234b6bdad, State: Initialized, Role: LEARNER
I20260926 03:44:36.017627 7755 consensus_queue.cc:260] T 00000000000000000000000000000000 P 037743be8b9845979457b84234b6bdad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "be6fefc6a90b41218affcf0df9459735" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 44939 } } peers { permanent_uuid: "af7004004c624669ae81014cf255d63f" member_type: VOTER last_known_addr { host: "127.0.194.253" port: 37485 } attrs { promote: false } } peers { permanent_uuid: "037743be8b9845979457b84234b6bdad" member_type: NON_VOTER last_known_addr { host: "127.0.194.252" port: 34225 } attrs { promote: true } }
I20260926 03:44:36.019703 7758 sys_catalog.cc:455] T 00000000000000000000000000000000 P 037743be8b9845979457b84234b6bdad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "be6fefc6a90b41218affcf0df9459735" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 44939 } } peers { permanent_uuid: "af7004004c624669ae81014cf255d63f" member_type: VOTER last_known_addr { host: "127.0.194.253" port: 37485 } attrs { promote: false } } peers { permanent_uuid: "037743be8b9845979457b84234b6bdad" member_type: NON_VOTER last_known_addr { host: "127.0.194.252" port: 34225 } attrs { promote: true } } }
I20260926 03:44:36.020030 7755 sys_catalog.cc:564] T 00000000000000000000000000000000 P 037743be8b9845979457b84234b6bdad [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:44:36.020139 7758 sys_catalog.cc:458] T 00000000000000000000000000000000 P 037743be8b9845979457b84234b6bdad [sys.catalog]: This master's current role is: LEARNER
I20260926 03:44:36.041016 7769 catalog_manager.cc:1259] Loaded cluster ID: 5900f64ad7ae4b009c7ea641aa872bad
I20260926 03:44:36.041158 7769 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 037743be8b9845979457b84234b6bdad: loading cluster ID for follower catalog manager: success
I20260926 03:44:36.043779 7769 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 037743be8b9845979457b84234b6bdad: acquiring CA information for follower catalog manager: success
I20260926 03:44:36.045521 7769 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 037743be8b9845979457b84234b6bdad: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260926 03:44:36.323966 7707 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:44:36.324272 7707 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:44:36.324380 7707 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:44:36.331898 7707 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:44:36.332468 7707 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:44:36.332698 7707 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.194.193
I20260926 03:44:36.343456 7707 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.194.212:35975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/krb5kdc/oryx_127.0.194.193.keytab
--principal=oryx/127.0.194.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.194.193:32927
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.194.193
--webserver_port=38631
--tserver_master_addrs=127.0.194.254:44939,127.0.194.253:37485,127.0.194.252:34225
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.194.193
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:44:36.344827 7707 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:44:36.346146 7707 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:44:36.360070 7775 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:44:36.360034 7778 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:44:36.359947 7776 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:44:36.361179 7707 server_base.cc:1034] running on GCE node
I20260926 03:44:36.361769 7707 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:44:36.362653 7707 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:44:36.363849 7707 hybrid_clock.cc:648] HybridClock initialized: now 1790394276363773 us; error 55 us; skew 500 ppm
I20260926 03:44:36.364171 7707 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:44:36 dist-test-slave-n2th krb5kdc[6888](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394276, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.194.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:44:36.367362 7707 init.cc:377] Logged in from keytab as oryx/127.0.194.193@KRBTEST.COM (short username oryx)
I20260926 03:44:36.368871 7707 webserver.cc:460] Webserver started at http://127.0.194.193:38631/ using document root <none> and password file <none>
I20260926 03:44:36.369544 7707 fs_manager.cc:359] Metadata directory not provided
I20260926 03:44:36.369657 7707 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:44:36.376437 7707 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 03:44:36.379482 7787 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:44:36.379657 7784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:36.372701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.252:34225 (local address 127.0.0.1:50470)
0926 03:44:36.373176 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0926 03:44:36.373202 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0926 03:44:36.373396 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:44:36.373826 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:44:36.373831 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:44:36.373856 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:44:36.374061 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:36.374072 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:36.376187 (+ 2115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:36.376193 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:36.377412 (+ 1219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:36.377425 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:36.378805 (+ 1380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:36.378811 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:36.378999 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:36.379043 (+ 44us) client_negotiation.cc:769] Sending connection context
0926 03:44:36.379154 (+ 111us) client_negotiation.cc:240] Negotiation successful
0926 03:44:36.379310 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":260,"threads_started":1}
I20260926 03:44:36.379671 7785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:36.372795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50470 (local address 127.0.194.252:34225)
0926 03:44:36.373490 (+ 695us) server_negotiation.cc:206] Beginning negotiation
0926 03:44:36.373498 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:44:36.373516 (+ 18us) server_negotiation.cc:407] Connection header received
0926 03:44:36.373582 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:44:36.373585 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:44:36.373652 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:44:36.373804 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:44:36.374202 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:36.376036 (+ 1834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:36.377566 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:36.378583 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:36.378736 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:36.378979 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0926 03:44:36.379179 (+ 200us) server_negotiation.cc:299] Negotiation successful
0926 03:44:36.379313 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":556,"thread_start_us":428,"threads_started":1}
I20260926 03:44:36.380594 7730 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 037743be8b9845979457b84234b6bdad [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260926 03:44:36.381232 7707 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260926 03:44:36.381202 7758 sys_catalog.cc:455] T 00000000000000000000000000000000 P 037743be8b9845979457b84234b6bdad [sys.catalog]: SysCatalogTable state changed. Reason: New leader be6fefc6a90b41218affcf0df9459735. Latest consensus state: current_term: 1 leader_uuid: "be6fefc6a90b41218affcf0df9459735" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "be6fefc6a90b41218affcf0df9459735" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 44939 } } peers { permanent_uuid: "af7004004c624669ae81014cf255d63f" member_type: VOTER last_known_addr { host: "127.0.194.253" port: 37485 } attrs { promote: false } } peers { permanent_uuid: "037743be8b9845979457b84234b6bdad" member_type: NON_VOTER last_known_addr { host: "127.0.194.252" port: 34225 } attrs { promote: true } } }
I20260926 03:44:36.381522 7758 sys_catalog.cc:458] T 00000000000000000000000000000000 P 037743be8b9845979457b84234b6bdad [sys.catalog]: This master's current role is: LEARNER
I20260926 03:44:36.381428 7707 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/minicluster-data/ts-0/wal
uuid: "b08e95643f1a4377b8c2d42ac8762a70"
format_stamp: "Formatted at 2026-09-26 03:44:33 on dist-test-slave-n2th"
server_key: "8c4e532c88a083b3efed8ae4b2f9f88e"
server_key_iv: "76aaae7dc44e31c11b365444fb467be4"
server_key_version: "encryptionkey@0"
I20260926 03:44:36.382534 7707 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:44:36.397567 7707 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:44:36.402052 7707 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:44:36.402271 7707 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:44:36.403240 7707 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:44:36.404572 7707 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:44:36.404657 7707 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:44:36.404727 7707 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:44:36.404780 7707 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:44:36.441458 7707 rpc_server.cc:307] RPC server started. Bound to: 127.0.194.193:32927
I20260926 03:44:36.441553 7899 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.194.193:32927 every 8 connection(s)
I20260926 03:44:36.442929 7707 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/minicluster-data/ts-0/data/info.pb
I20260926 03:44:36.451289 779 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu as pid 7707
I20260926 03:44:36.451639 779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu pid 7128
Sep 26 03:44:36 dist-test-slave-n2th krb5kdc[6888](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394276, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.194.193@KRBTEST.COM for oryx/127.0.194.252@KRBTEST.COM
Sep 26 03:44:36 dist-test-slave-n2th krb5kdc[6888](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394276, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.194.193@KRBTEST.COM for oryx/127.0.194.253@KRBTEST.COM
Sep 26 03:44:36 dist-test-slave-n2th krb5kdc[6888](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394276, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.194.193@KRBTEST.COM for oryx/127.0.194.254@KRBTEST.COM
Sep 26 03:44:36 dist-test-slave-n2th krb5kdc[6888](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394272, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.194.194@KRBTEST.COM
I20260926 03:44:36.471424 7785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:36.447554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.193:52475 (local address 127.0.194.252:34225)
0926 03:44:36.447809 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0926 03:44:36.447815 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:44:36.448495 (+ 680us) server_negotiation.cc:407] Connection header received
0926 03:44:36.451684 (+ 3189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:44:36.451689 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:44:36.451779 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:44:36.451851 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:44:36.454674 (+ 2823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:36.456267 (+ 1593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:36.458216 (+ 1949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:36.458883 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:36.458982 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:36.463862 (+ 4880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:44:36.463868 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:44:36.463875 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:44:36.463922 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:44:36.467633 (+ 3711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:36.468582 (+ 949us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:36.468588 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:36.468592 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:36.468731 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:36.469334 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:36.469339 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:36.469343 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:36.469688 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:44:36.469799 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0926 03:44:36.470850 (+ 1051us) server_negotiation.cc:299] Negotiation successful
0926 03:44:36.471083 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 03:44:36.473115 7908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:36.447538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.253:37485 (local address 127.0.194.193:51877)
0926 03:44:36.448641 (+ 1103us) negotiation.cc:107] Waiting for socket to connect
0926 03:44:36.448654 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 03:44:36.451360 (+ 2706us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:44:36.452283 (+ 923us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:44:36.452295 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:44:36.454029 (+ 1734us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:44:36.454463 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:36.454491 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:36.455643 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:36.455648 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:36.457882 (+ 2234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:36.457900 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:36.459181 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:36.459187 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:36.459516 (+ 329us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:36.460453 (+ 937us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:44:36.460479 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:44:36.464539 (+ 4060us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:44:36.468844 (+ 4305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:44:36.468851 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:44:36.468856 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:44:36.469381 (+ 525us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:44:36.469793 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:44:36.469797 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:44:36.469800 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:44:36.469957 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:44:36.470645 (+ 688us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:44:36.470652 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:44:36.470833 (+ 181us) client_negotiation.cc:769] Sending connection context
0926 03:44:36.471012 (+ 179us) client_negotiation.cc:240] Negotiation successful
0926 03:44:36.471582 (+ 570us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1007,"mutex_wait_us":34,"spinlock_wait_cycles":9600,"thread_start_us":649,"threads_started":1}
I20260926 03:44:36.473843 7904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:36.445823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.193:47353 (local address 127.0.194.254:44939)
0926 03:44:36.446879 (+ 1056us) server_negotiation.cc:206] Beginning negotiation
0926 03:44:36.446886 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:44:36.451406 (+ 4520us) server_negotiation.cc:407] Connection header received
0926 03:44:36.451693 (+ 287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:44:36.451699 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:44:36.451782 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:44:36.451867 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:44:36.456064 (+ 4197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:36.457181 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:36.459060 (+ 1879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:36.459669 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:36.459770 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:36.466341 (+ 6571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:44:36.466348 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:44:36.466353 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:44:36.466391 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:44:36.469969 (+ 3578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:36.471135 (+ 1166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:36.471140 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:36.471144 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:36.471290 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:36.471843 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:36.471848 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:36.471850 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:36.472618 (+ 768us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:44:36.472742 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0926 03:44:36.473286 (+ 544us) server_negotiation.cc:299] Negotiation successful
0926 03:44:36.473533 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":885,"thread_start_us":774,"threads_started":1}
I20260926 03:44:36.474442 7905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:36.446213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.193:51877 (local address 127.0.194.253:37485)
0926 03:44:36.447068 (+ 855us) server_negotiation.cc:206] Beginning negotiation
0926 03:44:36.447072 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:44:36.448748 (+ 1676us) server_negotiation.cc:407] Connection header received
0926 03:44:36.451763 (+ 3015us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:44:36.451768 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:44:36.451860 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:44:36.452573 (+ 713us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:44:36.454641 (+ 2068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:36.455488 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:36.458089 (+ 2601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:36.458705 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:36.458855 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:36.464812 (+ 5957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:44:36.464821 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:44:36.464828 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:44:36.464891 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:44:36.468665 (+ 3774us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:36.469513 (+ 848us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:36.469518 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:36.469522 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:36.469647 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:36.470076 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:36.470081 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:36.470084 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:36.470452 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:44:36.470577 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0926 03:44:36.473849 (+ 3272us) server_negotiation.cc:299] Negotiation successful
0926 03:44:36.474124 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":747,"thread_start_us":620,"threads_started":1}
I20260926 03:44:36.473136 7907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:36.446484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.252:34225 (local address 127.0.194.193:52475)
0926 03:44:36.448296 (+ 1812us) negotiation.cc:107] Waiting for socket to connect
0926 03:44:36.448350 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0926 03:44:36.451360 (+ 3010us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:44:36.452283 (+ 923us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:44:36.452295 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:44:36.453591 (+ 1296us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:44:36.454463 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:36.454494 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:36.456441 (+ 1947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:36.456448 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:36.458054 (+ 1606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:36.458067 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:36.459266 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:36.459272 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:36.459519 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:36.460401 (+ 882us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:44:36.460443 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:44:36.463595 (+ 3152us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:44:36.467798 (+ 4203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:44:36.467808 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:44:36.467839 (+ 31us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:44:36.468437 (+ 598us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:44:36.468867 (+ 430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:44:36.468871 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:44:36.468874 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:44:36.469185 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:44:36.469901 (+ 716us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:44:36.469910 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:44:36.470381 (+ 471us) client_negotiation.cc:769] Sending connection context
0926 03:44:36.471609 (+ 1228us) client_negotiation.cc:240] Negotiation successful
0926 03:44:36.471860 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1484,"mutex_wait_us":99,"spinlock_wait_cycles":23168,"thread_start_us":4353,"threads_started":1}
I20260926 03:44:36.474833 7906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:36.448014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:44939 (local address 127.0.194.193:47353)
0926 03:44:36.451288 (+ 3274us) negotiation.cc:107] Waiting for socket to connect
0926 03:44:36.451310 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 03:44:36.451524 (+ 214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:44:36.452307 (+ 783us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:44:36.452312 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:44:36.453413 (+ 1101us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:44:36.455877 (+ 2464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:36.455893 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:36.457356 (+ 1463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:36.457361 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:36.458904 (+ 1543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:36.458917 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:36.460574 (+ 1657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:36.460584 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:36.460780 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:36.461395 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:44:36.461426 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:44:36.466104 (+ 4678us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:44:36.470125 (+ 4021us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:44:36.470130 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:44:36.470133 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:44:36.470992 (+ 859us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:44:36.471450 (+ 458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:44:36.471456 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:44:36.471460 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:44:36.471714 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:44:36.472820 (+ 1106us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:44:36.472825 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:44:36.472978 (+ 153us) client_negotiation.cc:769] Sending connection context
0926 03:44:36.474086 (+ 1108us) client_negotiation.cc:240] Negotiation successful
0926 03:44:36.474357 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3139,"mutex_wait_us":89,"spinlock_wait_cycles":17792,"thread_start_us":976,"threads_started":1}
I20260926 03:44:36.475850 7902 heartbeater.cc:347] Connected to a master server at 127.0.194.253:37485
I20260926 03:44:36.475912 7900 heartbeater.cc:347] Connected to a master server at 127.0.194.252:34225
I20260926 03:44:36.475951 7901 heartbeater.cc:347] Connected to a master server at 127.0.194.254:44939
I20260926 03:44:36.476197 7900 heartbeater.cc:464] Registering TS with master...
I20260926 03:44:36.476203 7902 heartbeater.cc:464] Registering TS with master...
I20260926 03:44:36.476199 7901 heartbeater.cc:464] Registering TS with master...
I20260926 03:44:36.477155 7900 heartbeater.cc:511] Master 127.0.194.252:34225 requested a full tablet report, sending...
I20260926 03:44:36.477412 7902 heartbeater.cc:511] Master 127.0.194.253:37485 requested a full tablet report, sending...
I20260926 03:44:36.477756 7910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:36.453006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35854 (local address 127.0.194.194:42589)
0926 03:44:36.453709 (+ 703us) server_negotiation.cc:206] Beginning negotiation
0926 03:44:36.453718 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:44:36.453746 (+ 28us) server_negotiation.cc:407] Connection header received
0926 03:44:36.453809 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:44:36.453816 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:44:36.454055 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:44:36.454207 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:44:36.456673 (+ 2466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:36.458178 (+ 1505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:36.460005 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:36.460804 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:36.462638 (+ 1834us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:36.467698 (+ 5060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:44:36.467712 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:44:36.467730 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:44:36.467771 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:44:36.472348 (+ 4577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:36.473226 (+ 878us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:36.473235 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:36.473239 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:36.473376 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:36.473838 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:36.473843 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:36.473846 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:36.474243 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:44:36.474526 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0926 03:44:36.477077 (+ 2551us) server_negotiation.cc:299] Negotiation successful
0926 03:44:36.477349 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":560,"thread_start_us":411,"threads_started":1}
I20260926 03:44:36.477872 7901 heartbeater.cc:511] Master 127.0.194.254:44939 requested a full tablet report, sending...
I20260926 03:44:36.478966 6933 ts_manager.cc:194] Re-registered known tserver with Master: b08e95643f1a4377b8c2d42ac8762a70 (127.0.194.193:32927)
I20260926 03:44:36.479103 7251 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:44:36.479609 7720 ts_manager.cc:194] Registered new tserver with Master: b08e95643f1a4377b8c2d42ac8762a70 (127.0.194.193:32927)
I20260926 03:44:36.480015 6933 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.194.193@KRBTEST.COM'} at 127.0.194.193:47353
I20260926 03:44:36.480321 7531 ts_manager.cc:194] Registered new tserver with Master: b08e95643f1a4377b8c2d42ac8762a70 (127.0.194.193:32927)
I20260926 03:44:36.565428 779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu with pid 7128
I20260926 03:44:36.581627 779 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
/tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.194.194:42589
--local_ip_for_outbound_sockets=127.0.194.194
--tserver_master_addrs=127.0.194.254:44939
--webserver_port=35031
--webserver_interface=127.0.194.194
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/krb5kdc/oryx_127.0.194.194.keytab
--principal=oryx/127.0.194.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.194.212:35975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.194.254:44939,127.0.194.253:37485
--tserver_master_addrs=127.0.194.254:44939,127.0.194.253:37485,127.0.194.252:34225 with env {KRB5CCNAME=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 03:44:36.847157 7773 sys_catalog.cc:455] T 00000000000000000000000000000000 P be6fefc6a90b41218affcf0df9459735 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "be6fefc6a90b41218affcf0df9459735" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "be6fefc6a90b41218affcf0df9459735" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 44939 } } peers { permanent_uuid: "af7004004c624669ae81014cf255d63f" member_type: VOTER last_known_addr { host: "127.0.194.253" port: 37485 } attrs { promote: false } } peers { permanent_uuid: "037743be8b9845979457b84234b6bdad" member_type: NON_VOTER last_known_addr { host: "127.0.194.252" port: 34225 } attrs { promote: true } } }
I20260926 03:44:36.847554 7773 sys_catalog.cc:458] T 00000000000000000000000000000000 P be6fefc6a90b41218affcf0df9459735 [sys.catalog]: This master's current role is: LEADER
I20260926 03:44:36.848848 7773 raft_consensus.cc:1050] T 00000000000000000000000000000000 P be6fefc6a90b41218affcf0df9459735: attempting to promote NON_VOTER 037743be8b9845979457b84234b6bdad to VOTER
I20260926 03:44:36.849392 7773 consensus_queue.cc:237] T 00000000000000000000000000000000 P be6fefc6a90b41218affcf0df9459735 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "be6fefc6a90b41218affcf0df9459735" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 44939 } } peers { permanent_uuid: "af7004004c624669ae81014cf255d63f" member_type: VOTER last_known_addr { host: "127.0.194.253" port: 37485 } attrs { promote: false } } peers { permanent_uuid: "037743be8b9845979457b84234b6bdad" member_type: VOTER last_known_addr { host: "127.0.194.252" port: 34225 } attrs { promote: false } }
I20260926 03:44:36.851627 7730 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 037743be8b9845979457b84234b6bdad [term 1 LEARNER]: Refusing update from remote peer be6fefc6a90b41218affcf0df9459735: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260926 03:44:36.851663 7541 raft_consensus.cc:1261] T 00000000000000000000000000000000 P af7004004c624669ae81014cf255d63f [term 1 FOLLOWER]: Refusing update from remote peer be6fefc6a90b41218affcf0df9459735: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260926 03:44:36.852252 7914 consensus_queue.cc:1035] T 00000000000000000000000000000000 P be6fefc6a90b41218affcf0df9459735 [LEADER]: Connected to new peer: Peer: permanent_uuid: "037743be8b9845979457b84234b6bdad" member_type: VOTER last_known_addr { host: "127.0.194.252" port: 34225 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260926 03:44:36.852710 7773 consensus_queue.cc:1035] T 00000000000000000000000000000000 P be6fefc6a90b41218affcf0df9459735 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af7004004c624669ae81014cf255d63f" member_type: VOTER last_known_addr { host: "127.0.194.253" port: 37485 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260926 03:44:36.855906 7913 raft_consensus.cc:2932] T 00000000000000000000000000000000 P be6fefc6a90b41218affcf0df9459735 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 037743be8b9845979457b84234b6bdad (127.0.194.252) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "be6fefc6a90b41218affcf0df9459735" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 44939 } } peers { permanent_uuid: "af7004004c624669ae81014cf255d63f" member_type: VOTER last_known_addr { host: "127.0.194.253" port: 37485 } attrs { promote: false } } peers { permanent_uuid: "037743be8b9845979457b84234b6bdad" member_type: VOTER last_known_addr { host: "127.0.194.252" port: 34225 } attrs { promote: false } } }
I20260926 03:44:36.856539 7541 raft_consensus.cc:2932] T 00000000000000000000000000000000 P af7004004c624669ae81014cf255d63f [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 037743be8b9845979457b84234b6bdad (127.0.194.252) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "be6fefc6a90b41218affcf0df9459735" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 44939 } } peers { permanent_uuid: "af7004004c624669ae81014cf255d63f" member_type: VOTER last_known_addr { host: "127.0.194.253" port: 37485 } attrs { promote: false } } peers { permanent_uuid: "037743be8b9845979457b84234b6bdad" member_type: VOTER last_known_addr { host: "127.0.194.252" port: 34225 } attrs { promote: false } } }
I20260926 03:44:36.859763 7913 sys_catalog.cc:455] T 00000000000000000000000000000000 P be6fefc6a90b41218affcf0df9459735 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "be6fefc6a90b41218affcf0df9459735" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "be6fefc6a90b41218affcf0df9459735" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 44939 } } peers { permanent_uuid: "af7004004c624669ae81014cf255d63f" member_type: VOTER last_known_addr { host: "127.0.194.253" port: 37485 } attrs { promote: false } } peers { permanent_uuid: "037743be8b9845979457b84234b6bdad" member_type: VOTER last_known_addr { host: "127.0.194.252" port: 34225 } attrs { promote: false } } }
I20260926 03:44:36.860177 7913 sys_catalog.cc:458] T 00000000000000000000000000000000 P be6fefc6a90b41218affcf0df9459735 [sys.catalog]: This master's current role is: LEADER
I20260926 03:44:36.860532 7773 sys_catalog.cc:455] T 00000000000000000000000000000000 P be6fefc6a90b41218affcf0df9459735 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "be6fefc6a90b41218affcf0df9459735" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "be6fefc6a90b41218affcf0df9459735" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 44939 } } peers { permanent_uuid: "af7004004c624669ae81014cf255d63f" member_type: VOTER last_known_addr { host: "127.0.194.253" port: 37485 } attrs { promote: false } } peers { permanent_uuid: "037743be8b9845979457b84234b6bdad" member_type: VOTER last_known_addr { host: "127.0.194.252" port: 34225 } attrs { promote: false } } }
I20260926 03:44:36.860786 7773 sys_catalog.cc:458] T 00000000000000000000000000000000 P be6fefc6a90b41218affcf0df9459735 [sys.catalog]: This master's current role is: LEADER
I20260926 03:44:36.863787 7773 sys_catalog.cc:455] T 00000000000000000000000000000000 P be6fefc6a90b41218affcf0df9459735 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "be6fefc6a90b41218affcf0df9459735" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "be6fefc6a90b41218affcf0df9459735" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 44939 } } peers { permanent_uuid: "af7004004c624669ae81014cf255d63f" member_type: VOTER last_known_addr { host: "127.0.194.253" port: 37485 } attrs { promote: false } } peers { permanent_uuid: "037743be8b9845979457b84234b6bdad" member_type: VOTER last_known_addr { host: "127.0.194.252" port: 34225 } attrs { promote: false } } }
I20260926 03:44:36.864173 7773 sys_catalog.cc:458] T 00000000000000000000000000000000 P be6fefc6a90b41218affcf0df9459735 [sys.catalog]: This master's current role is: LEADER
I20260926 03:44:36.865396 7730 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 037743be8b9845979457b84234b6bdad [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 037743be8b9845979457b84234b6bdad (127.0.194.252) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "be6fefc6a90b41218affcf0df9459735" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 44939 } } peers { permanent_uuid: "af7004004c624669ae81014cf255d63f" member_type: VOTER last_known_addr { host: "127.0.194.253" port: 37485 } attrs { promote: false } } peers { permanent_uuid: "037743be8b9845979457b84234b6bdad" member_type: VOTER last_known_addr { host: "127.0.194.252" port: 34225 } attrs { promote: false } } }
I20260926 03:44:36.868884 7919 sys_catalog.cc:455] T 00000000000000000000000000000000 P af7004004c624669ae81014cf255d63f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "be6fefc6a90b41218affcf0df9459735" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "be6fefc6a90b41218affcf0df9459735" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 44939 } } peers { permanent_uuid: "af7004004c624669ae81014cf255d63f" member_type: VOTER last_known_addr { host: "127.0.194.253" port: 37485 } attrs { promote: false } } peers { permanent_uuid: "037743be8b9845979457b84234b6bdad" member_type: VOTER last_known_addr { host: "127.0.194.252" port: 34225 } attrs { promote: false } } }
I20260926 03:44:36.869342 7919 sys_catalog.cc:458] T 00000000000000000000000000000000 P af7004004c624669ae81014cf255d63f [sys.catalog]: This master's current role is: FOLLOWER
I20260926 03:44:36.870175 7758 sys_catalog.cc:455] T 00000000000000000000000000000000 P 037743be8b9845979457b84234b6bdad [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "be6fefc6a90b41218affcf0df9459735" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "be6fefc6a90b41218affcf0df9459735" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 44939 } } peers { permanent_uuid: "af7004004c624669ae81014cf255d63f" member_type: VOTER last_known_addr { host: "127.0.194.253" port: 37485 } attrs { promote: false } } peers { permanent_uuid: "037743be8b9845979457b84234b6bdad" member_type: VOTER last_known_addr { host: "127.0.194.252" port: 34225 } attrs { promote: false } } }
I20260926 03:44:36.870426 7758 sys_catalog.cc:458] T 00000000000000000000000000000000 P 037743be8b9845979457b84234b6bdad [sys.catalog]: This master's current role is: FOLLOWER
W20260926 03:44:36.920507 7912 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:44:36.920805 7912 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:44:36.921000 7912 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:44:36.928678 7912 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:44:36.928805 7912 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:44:36.928920 7912 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.194.194
I20260926 03:44:36.939553 7912 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.194.212:35975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/krb5kdc/oryx_127.0.194.194.keytab
--principal=oryx/127.0.194.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.194.194:42589
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.194.194
--webserver_port=35031
--tserver_master_addrs=127.0.194.254:44939,127.0.194.253:37485,127.0.194.252:34225
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.194.194
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:44:36.941059 7912 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:44:36.942535 7912 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:44:36.955634 7924 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:44:36.955641 7927 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:44:36.955678 7925 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:44:36.956543 7912 server_base.cc:1034] running on GCE node
I20260926 03:44:36.957151 7912 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:44:36.958113 7912 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:44:36.959368 7912 hybrid_clock.cc:648] HybridClock initialized: now 1790394276959314 us; error 68 us; skew 500 ppm
I20260926 03:44:36.959713 7912 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:44:36 dist-test-slave-n2th krb5kdc[6888](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394276, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.194.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:44:36.962832 7912 init.cc:377] Logged in from keytab as oryx/127.0.194.194@KRBTEST.COM (short username oryx)
I20260926 03:44:36.964316 7912 webserver.cc:460] Webserver started at http://127.0.194.194:35031/ using document root <none> and password file <none>
I20260926 03:44:36.965011 7912 fs_manager.cc:359] Metadata directory not provided
I20260926 03:44:36.965178 7912 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:44:36.970570 7912 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260926 03:44:36.973409 7934 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:44:36.974891 7912 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 03:44:36.975070 7912 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/minicluster-data/ts-1/wal
uuid: "949d66f722dd4b5c86409272002b87b3"
format_stamp: "Formatted at 2026-09-26 03:44:33 on dist-test-slave-n2th"
server_key: "aaf9511636928b1f9d1b93be37657084"
server_key_iv: "0a09fee6cac4ecc693509f989c1062c2"
server_key_version: "encryptionkey@0"
I20260926 03:44:36.976187 7912 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:44:37.010172 7912 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:44:37.014655 7912 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:44:37.014977 7912 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:44:37.015969 7912 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:44:37.017313 7912 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:44:37.017449 7912 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:44:37.017584 7912 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:44:37.017678 7912 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:44:37.052214 7912 rpc_server.cc:307] RPC server started. Bound to: 127.0.194.194:42589
I20260926 03:44:37.052275 8046 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.194.194:42589 every 8 connection(s)
I20260926 03:44:37.053682 7912 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/minicluster-data/ts-1/data/info.pb
I20260926 03:44:37.062971 779 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu as pid 7912
I20260926 03:44:37.063262 779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu pid 7261
Sep 26 03:44:37 dist-test-slave-n2th krb5kdc[6888](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394272, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.194.195@KRBTEST.COM
Sep 26 03:44:37 dist-test-slave-n2th krb5kdc[6888](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394276, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.194.194@KRBTEST.COM for oryx/127.0.194.253@KRBTEST.COM
Sep 26 03:44:37 dist-test-slave-n2th krb5kdc[6888](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394276, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.194.194@KRBTEST.COM for oryx/127.0.194.254@KRBTEST.COM
Sep 26 03:44:37 dist-test-slave-n2th krb5kdc[6888](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394276, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.194.194@KRBTEST.COM for oryx/127.0.194.252@KRBTEST.COM
I20260926 03:44:37.090540 8058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:37.064508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41214 (local address 127.0.194.195:34367)
0926 03:44:37.067232 (+ 2724us) server_negotiation.cc:206] Beginning negotiation
0926 03:44:37.067247 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 03:44:37.067273 (+ 26us) server_negotiation.cc:407] Connection header received
0926 03:44:37.067330 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:44:37.067342 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:44:37.067538 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:44:37.067713 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:44:37.069326 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:37.071046 (+ 1720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:37.073430 (+ 2384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:37.074291 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:37.074479 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:37.081864 (+ 7385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:44:37.081877 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:44:37.081891 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:44:37.081924 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:44:37.085844 (+ 3920us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:37.086798 (+ 954us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:37.086810 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:37.086816 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:37.086998 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:37.087509 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:37.087514 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:37.087518 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:37.087952 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:44:37.088240 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0926 03:44:37.089820 (+ 1580us) server_negotiation.cc:299] Negotiation successful
0926 03:44:37.090103 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2561,"thread_start_us":2419,"threads_started":1}
I20260926 03:44:37.092012 8051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:37.056672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.194:48229 (local address 127.0.194.252:34225)
0926 03:44:37.068724 (+ 12052us) server_negotiation.cc:206] Beginning negotiation
0926 03:44:37.068734 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 03:44:37.068758 (+ 24us) server_negotiation.cc:407] Connection header received
0926 03:44:37.068826 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:44:37.068833 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:44:37.068923 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:44:37.069057 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:44:37.072498 (+ 3441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:37.073677 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:37.076350 (+ 2673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:37.077000 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:37.077121 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:37.084962 (+ 7841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:44:37.084969 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:44:37.084974 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:44:37.085014 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:44:37.088455 (+ 3441us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:37.089287 (+ 832us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:37.089293 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:37.089297 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:37.089433 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:37.090057 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:37.090064 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:37.090067 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:37.090550 (+ 483us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:44:37.090766 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0926 03:44:37.091524 (+ 758us) server_negotiation.cc:299] Negotiation successful
0926 03:44:37.091746 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11887,"thread_start_us":12514,"threads_started":1}
I20260926 03:44:37.092031 7384 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:44:37.093408 8054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:37.058586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.194:53851 (local address 127.0.194.254:44939)
0926 03:44:37.069735 (+ 11149us) server_negotiation.cc:206] Beginning negotiation
0926 03:44:37.069744 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:44:37.069770 (+ 26us) server_negotiation.cc:407] Connection header received
0926 03:44:37.069847 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:44:37.069855 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:44:37.069955 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:44:37.070103 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:44:37.072540 (+ 2437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:37.073562 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:37.076807 (+ 3245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:37.077544 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:37.078408 (+ 864us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:37.083251 (+ 4843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:44:37.083258 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:44:37.083265 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:44:37.083310 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:44:37.087518 (+ 4208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:37.088865 (+ 1347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:37.088871 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:37.088874 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:37.089038 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:37.089456 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:37.089461 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:37.089463 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:37.089828 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:44:37.089980 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0926 03:44:37.092572 (+ 2592us) server_negotiation.cc:299] Negotiation successful
0926 03:44:37.092891 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10989,"thread_start_us":10883,"threads_started":1}
I20260926 03:44:37.095649 8056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:37.057409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.252:34225 (local address 127.0.194.194:48229)
0926 03:44:37.065782 (+ 8373us) negotiation.cc:107] Waiting for socket to connect
0926 03:44:37.065806 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 03:44:37.065951 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:44:37.069433 (+ 3482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:44:37.069449 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:44:37.071420 (+ 1971us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:44:37.072252 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:37.072284 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:37.073833 (+ 1549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:37.073837 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:37.076143 (+ 2306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:37.076165 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:37.077425 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:37.077432 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:37.077784 (+ 352us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:37.078736 (+ 952us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:44:37.078784 (+ 48us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:44:37.084685 (+ 5901us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:44:37.088591 (+ 3906us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:44:37.088594 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:44:37.088597 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:44:37.089143 (+ 546us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:44:37.089569 (+ 426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:44:37.089574 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:44:37.089577 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:44:37.089856 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:44:37.090862 (+ 1006us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:44:37.090866 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:44:37.091062 (+ 196us) client_negotiation.cc:769] Sending connection context
0926 03:44:37.091221 (+ 159us) client_negotiation.cc:240] Negotiation successful
0926 03:44:37.094122 (+ 2901us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8206,"mutex_wait_us":5301,"spinlock_wait_cycles":5888,"thread_start_us":5100,"threads_started":1}
I20260926 03:44:37.095774 8053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:37.058236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.194:60235 (local address 127.0.194.253:37485)
0926 03:44:37.070585 (+ 12349us) server_negotiation.cc:206] Beginning negotiation
0926 03:44:37.070589 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:44:37.070604 (+ 15us) server_negotiation.cc:407] Connection header received
0926 03:44:37.070660 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:44:37.070663 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:44:37.070723 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:44:37.070901 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:44:37.072480 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:37.073652 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:37.076625 (+ 2973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:37.077279 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:37.077403 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:37.082936 (+ 5533us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:44:37.082941 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:44:37.082944 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:44:37.082985 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:44:37.086698 (+ 3713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:37.087755 (+ 1057us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:37.087761 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:37.087764 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:37.087922 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:37.088864 (+ 942us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:37.088869 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:37.088872 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:37.089272 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:44:37.089420 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0926 03:44:37.091510 (+ 2090us) server_negotiation.cc:299] Negotiation successful
0926 03:44:37.095331 (+ 3821us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12242,"thread_start_us":12132,"threads_started":1}
I20260926 03:44:37.095762 8055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:37.057374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:44939 (local address 127.0.194.194:53851)
0926 03:44:37.060771 (+ 3397us) negotiation.cc:107] Waiting for socket to connect
0926 03:44:37.060790 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 03:44:37.062179 (+ 1389us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:44:37.070120 (+ 7941us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:44:37.070128 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:44:37.070791 (+ 663us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:44:37.072262 (+ 1471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:37.072284 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:37.073733 (+ 1449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:37.073738 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:37.076608 (+ 2870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:37.076626 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:37.077791 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:37.077798 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:37.078072 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:37.079002 (+ 930us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:44:37.079026 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:44:37.082998 (+ 3972us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:44:37.087698 (+ 4700us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:44:37.087703 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:44:37.087706 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:44:37.088707 (+ 1001us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:44:37.089174 (+ 467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:44:37.089179 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:44:37.089183 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:44:37.089334 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:44:37.090047 (+ 713us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:44:37.090053 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:44:37.090518 (+ 465us) client_negotiation.cc:769] Sending connection context
0926 03:44:37.091222 (+ 704us) client_negotiation.cc:240] Negotiation successful
0926 03:44:37.091774 (+ 552us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3214,"mutex_wait_us":3500,"spinlock_wait_cycles":1280,"thread_start_us":1804,"threads_started":1}
I20260926 03:44:37.096689 8052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:37.057374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.253:37485 (local address 127.0.194.194:60235)
0926 03:44:37.058483 (+ 1109us) negotiation.cc:107] Waiting for socket to connect
0926 03:44:37.058544 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0926 03:44:37.061855 (+ 3311us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:44:37.070929 (+ 9074us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:44:37.070934 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:44:37.071500 (+ 566us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:44:37.072290 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:37.072299 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:37.073841 (+ 1542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:37.073849 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:37.076462 (+ 2613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:37.076472 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:37.077651 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:37.077657 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:37.077904 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:37.078815 (+ 911us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:44:37.078833 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:44:37.082597 (+ 3764us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:44:37.086945 (+ 4348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:44:37.086957 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:44:37.086978 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:44:37.087613 (+ 635us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:44:37.088075 (+ 462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:44:37.088082 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:44:37.088088 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:44:37.088741 (+ 653us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:44:37.089483 (+ 742us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:44:37.089493 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:44:37.091074 (+ 1581us) client_negotiation.cc:769] Sending connection context
0926 03:44:37.091222 (+ 148us) client_negotiation.cc:240] Negotiation successful
0926 03:44:37.096260 (+ 5038us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":724,"mutex_wait_us":5460,"spinlock_wait_cycles":62208,"thread_start_us":7554,"threads_started":1}
I20260926 03:44:37.097923 8048 heartbeater.cc:347] Connected to a master server at 127.0.194.254:44939
I20260926 03:44:37.098088 8049 heartbeater.cc:347] Connected to a master server at 127.0.194.253:37485
I20260926 03:44:37.098263 8049 heartbeater.cc:464] Registering TS with master...
I20260926 03:44:37.098268 8048 heartbeater.cc:464] Registering TS with master...
I20260926 03:44:37.098321 8047 heartbeater.cc:347] Connected to a master server at 127.0.194.252:34225
I20260926 03:44:37.098482 8047 heartbeater.cc:464] Registering TS with master...
I20260926 03:44:37.099577 8049 heartbeater.cc:511] Master 127.0.194.253:37485 requested a full tablet report, sending...
I20260926 03:44:37.099586 8047 heartbeater.cc:511] Master 127.0.194.252:34225 requested a full tablet report, sending...
I20260926 03:44:37.099644 8048 heartbeater.cc:511] Master 127.0.194.254:44939 requested a full tablet report, sending...
I20260926 03:44:37.101462 6933 ts_manager.cc:194] Re-registered known tserver with Master: 949d66f722dd4b5c86409272002b87b3 (127.0.194.194:42589)
I20260926 03:44:37.102473 6933 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.194.194@KRBTEST.COM'} at 127.0.194.194:53851
I20260926 03:44:37.102795 7531 ts_manager.cc:194] Registered new tserver with Master: 949d66f722dd4b5c86409272002b87b3 (127.0.194.194:42589)
I20260926 03:44:37.103340 7720 ts_manager.cc:194] Registered new tserver with Master: 949d66f722dd4b5c86409272002b87b3 (127.0.194.194:42589)
I20260926 03:44:37.176539 779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu with pid 7261
I20260926 03:44:37.192687 779 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
/tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.194.195:34367
--local_ip_for_outbound_sockets=127.0.194.195
--tserver_master_addrs=127.0.194.254:44939
--webserver_port=32783
--webserver_interface=127.0.194.195
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/krb5kdc/oryx_127.0.194.195.keytab
--principal=oryx/127.0.194.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.194.212:35975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.194.254:44939,127.0.194.253:37485
--tserver_master_addrs=127.0.194.254:44939,127.0.194.253:37485,127.0.194.252:34225 with env {KRB5CCNAME=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 03:44:37.483458 7901 heartbeater.cc:503] Master 127.0.194.254:44939 was elected leader, sending a full tablet report...
W20260926 03:44:37.531894 8060 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:44:37.532174 8060 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:44:37.532266 8060 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:44:37.539606 8060 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:44:37.539738 8060 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:44:37.539865 8060 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.194.195
I20260926 03:44:37.549811 8060 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.194.212:35975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/krb5kdc/oryx_127.0.194.195.keytab
--principal=oryx/127.0.194.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.194.195:34367
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.194.195
--webserver_port=32783
--tserver_master_addrs=127.0.194.254:44939,127.0.194.253:37485,127.0.194.252:34225
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.194.195
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:44:37.551203 8060 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:44:37.552597 8060 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:44:37.566067 8068 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:44:37.566121 8066 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:44:37.566149 8065 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:44:37.567067 8060 server_base.cc:1034] running on GCE node
I20260926 03:44:37.567790 8060 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:44:37.568725 8060 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:44:37.569994 8060 hybrid_clock.cc:648] HybridClock initialized: now 1790394277569944 us; error 62 us; skew 500 ppm
I20260926 03:44:37.570353 8060 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:44:37 dist-test-slave-n2th krb5kdc[6888](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394277, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.194.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:44:37.573581 8060 init.cc:377] Logged in from keytab as oryx/127.0.194.195@KRBTEST.COM (short username oryx)
I20260926 03:44:37.575070 8060 webserver.cc:460] Webserver started at http://127.0.194.195:32783/ using document root <none> and password file <none>
I20260926 03:44:37.575753 8060 fs_manager.cc:359] Metadata directory not provided
I20260926 03:44:37.575906 8060 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:44:37.581136 8060 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.003s
I20260926 03:44:37.583949 8075 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:44:37.585326 8060 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 03:44:37.585517 8060 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/minicluster-data/ts-2/wal
uuid: "162b5f9a168f49db9ec3b1a5b0f0a526"
format_stamp: "Formatted at 2026-09-26 03:44:34 on dist-test-slave-n2th"
server_key: "ee8152e098a63fcbbadaa07bd1ffa5f1"
server_key_iv: "2984b4c524cc32929e9033fc7c332814"
server_key_version: "encryptionkey@0"
I20260926 03:44:37.586526 8060 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:44:37.609604 8060 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:44:37.613960 8060 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:44:37.614260 8060 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:44:37.615309 8060 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:44:37.616695 8060 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:44:37.616811 8060 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:44:37.616914 8060 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:44:37.616994 8060 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:44:37.650097 8060 rpc_server.cc:307] RPC server started. Bound to: 127.0.194.195:34367
I20260926 03:44:37.650188 8187 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.194.195:34367 every 8 connection(s)
I20260926 03:44:37.651600 8060 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/minicluster-data/ts-2/data/info.pb
I20260926 03:44:37.654320 779 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu as pid 8060
Sep 26 03:44:37 dist-test-slave-n2th krb5kdc[6888](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394277, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.194.195@KRBTEST.COM for oryx/127.0.194.253@KRBTEST.COM
Sep 26 03:44:37 dist-test-slave-n2th krb5kdc[6888](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394277, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.194.195@KRBTEST.COM for oryx/127.0.194.252@KRBTEST.COM
I20260926 03:44:37.723637 8195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:37.656394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.195:60217 (local address 127.0.194.253:37485)
0926 03:44:37.676426 (+ 20032us) server_negotiation.cc:206] Beginning negotiation
0926 03:44:37.676435 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:44:37.676459 (+ 24us) server_negotiation.cc:407] Connection header received
0926 03:44:37.676543 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:44:37.676551 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:44:37.676644 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:44:37.676775 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:44:37.679342 (+ 2567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:37.680681 (+ 1339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:37.701636 (+ 20955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:37.702608 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:37.702833 (+ 225us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:37.710802 (+ 7969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:44:37.710809 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:44:37.710819 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:44:37.710871 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:44:37.714683 (+ 3812us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:37.720694 (+ 6011us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:37.720699 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:37.720702 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:37.720858 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:37.721319 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:37.721324 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:37.721327 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:37.721673 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:44:37.721781 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0926 03:44:37.722614 (+ 833us) server_negotiation.cc:299] Negotiation successful
0926 03:44:37.723167 (+ 553us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19873,"thread_start_us":19823,"threads_started":1}
I20260926 03:44:37.723768 8198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:37.671782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.195:51233 (local address 127.0.194.252:34225)
0926 03:44:37.673907 (+ 2125us) server_negotiation.cc:206] Beginning negotiation
0926 03:44:37.673916 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:44:37.674585 (+ 669us) server_negotiation.cc:407] Connection header received
0926 03:44:37.674828 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:44:37.674835 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:44:37.674949 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:44:37.676594 (+ 1645us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:44:37.681251 (+ 4657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:37.682804 (+ 1553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:37.703504 (+ 20700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:37.704205 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:37.704362 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:37.711559 (+ 7197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:44:37.711565 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:44:37.711570 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:44:37.711610 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:44:37.717328 (+ 5718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:37.719451 (+ 2123us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:37.719456 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:37.719459 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:37.719605 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:37.720172 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:37.720175 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:37.720177 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:37.720442 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:44:37.720604 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0926 03:44:37.722982 (+ 2378us) server_negotiation.cc:299] Negotiation successful
0926 03:44:37.723250 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1956,"thread_start_us":1832,"threads_started":1}
I20260926 03:44:37.725410 8197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:37.656406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.253:37485 (local address 127.0.194.195:60217)
0926 03:44:37.672652 (+ 16246us) negotiation.cc:107] Waiting for socket to connect
0926 03:44:37.672686 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0926 03:44:37.672857 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:44:37.676929 (+ 4072us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:44:37.676933 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:44:37.677472 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:44:37.679178 (+ 1706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:37.679189 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:37.680855 (+ 1666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:37.680864 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:37.701349 (+ 20485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:37.701386 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:37.705556 (+ 4170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:37.705564 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:37.705737 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:37.706865 (+ 1128us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:44:37.706890 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:44:37.710561 (+ 3671us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:44:37.718503 (+ 7942us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:44:37.718509 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:44:37.718513 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:44:37.719061 (+ 548us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:44:37.721013 (+ 1952us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:44:37.721018 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:44:37.721020 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:44:37.721187 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:44:37.721865 (+ 678us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:44:37.721869 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:44:37.722007 (+ 138us) client_negotiation.cc:769] Sending connection context
0926 03:44:37.722191 (+ 184us) client_negotiation.cc:240] Negotiation successful
0926 03:44:37.722424 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16048,"mutex_wait_us":1273,"thread_start_us":17719,"threads_started":1}
I20260926 03:44:37.727056 8190 heartbeater.cc:347] Connected to a master server at 127.0.194.253:37485
I20260926 03:44:37.727314 8196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:37.671192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.252:34225 (local address 127.0.194.195:51233)
0926 03:44:37.674459 (+ 3267us) negotiation.cc:107] Waiting for socket to connect
0926 03:44:37.674483 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 03:44:37.674655 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:44:37.675488 (+ 833us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:44:37.675497 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:44:37.676492 (+ 995us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:44:37.681031 (+ 4539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:37.681051 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:37.683030 (+ 1979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:37.683039 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:37.703268 (+ 20229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:37.703289 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:37.704691 (+ 1402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:37.704699 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:37.705050 (+ 351us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:37.706131 (+ 1081us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:44:37.706174 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:44:37.711351 (+ 5177us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:44:37.717612 (+ 6261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:44:37.717624 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:44:37.717648 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:44:37.719311 (+ 1663us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:44:37.719752 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:44:37.719760 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:44:37.719763 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:44:37.720029 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:44:37.720584 (+ 555us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:44:37.720592 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:44:37.721229 (+ 637us) client_negotiation.cc:769] Sending connection context
0926 03:44:37.721566 (+ 337us) client_negotiation.cc:240] Negotiation successful
0926 03:44:37.722121 (+ 555us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3112,"mutex_wait_us":984,"thread_start_us":999,"threads_started":1}
I20260926 03:44:37.727392 8190 heartbeater.cc:464] Registering TS with master...
I20260926 03:44:37.728961 8188 heartbeater.cc:347] Connected to a master server at 127.0.194.252:34225
I20260926 03:44:37.729097 8188 heartbeater.cc:464] Registering TS with master...
I20260926 03:44:37.729318 8190 heartbeater.cc:511] Master 127.0.194.253:37485 requested a full tablet report, sending...
I20260926 03:44:37.729557 8188 heartbeater.cc:511] Master 127.0.194.252:34225 requested a full tablet report, sending...
I20260926 03:44:37.731225 7531 ts_manager.cc:194] Registered new tserver with Master: 162b5f9a168f49db9ec3b1a5b0f0a526 (127.0.194.195:34367)
I20260926 03:44:37.732424 8206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:37.683460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43918 (local address 127.0.194.253:37485)
0926 03:44:37.705115 (+ 21655us) server_negotiation.cc:206] Beginning negotiation
0926 03:44:37.705124 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:44:37.707012 (+ 1888us) server_negotiation.cc:407] Connection header received
0926 03:44:37.707324 (+ 312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:44:37.707329 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:44:37.707416 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:44:37.707522 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:44:37.710127 (+ 2605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:37.711580 (+ 1453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:37.716315 (+ 4735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:37.717010 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:37.717115 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:37.723178 (+ 6063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:44:37.723186 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:44:37.723191 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:44:37.723226 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:44:37.727344 (+ 4118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:37.728151 (+ 807us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:37.728156 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:37.728160 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:37.728305 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:37.729322 (+ 1017us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:37.729328 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:37.729331 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:37.730515 (+ 1184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:44:37.730799 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0926 03:44:37.731822 (+ 1023us) server_negotiation.cc:299] Negotiation successful
0926 03:44:37.732039 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21467,"thread_start_us":21701,"threads_started":1}
I20260926 03:44:37.732558 7720 ts_manager.cc:194] Registered new tserver with Master: 162b5f9a168f49db9ec3b1a5b0f0a526 (127.0.194.195:34367)
I20260926 03:44:37.734663 8205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:37.681079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53026 (local address 127.0.194.252:34225)
0926 03:44:37.703757 (+ 22678us) server_negotiation.cc:206] Beginning negotiation
0926 03:44:37.703763 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:44:37.704666 (+ 903us) server_negotiation.cc:407] Connection header received
0926 03:44:37.704721 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:44:37.704725 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:44:37.704799 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:44:37.704894 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:44:37.709271 (+ 4377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:37.710595 (+ 1324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:37.713059 (+ 2464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:37.714066 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:37.714227 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:37.722682 (+ 8455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:44:37.722688 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:44:37.722692 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:44:37.722725 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:44:37.727381 (+ 4656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:37.730907 (+ 3526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:37.730912 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:37.730916 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:37.731077 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:37.732928 (+ 1851us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:37.732933 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:37.732937 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:37.733290 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:44:37.733525 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0926 03:44:37.734091 (+ 566us) server_negotiation.cc:299] Negotiation successful
0926 03:44:37.734314 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22537,"thread_start_us":22692,"threads_started":1}
Sep 26 03:44:37 dist-test-slave-n2th krb5kdc[6888](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394277, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.194.195@KRBTEST.COM for oryx/127.0.194.254@KRBTEST.COM
I20260926 03:44:37.736554 8203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:37.679691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57254 (local address 127.0.194.254:44939)
0926 03:44:37.705801 (+ 26110us) server_negotiation.cc:206] Beginning negotiation
0926 03:44:37.705808 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:44:37.706865 (+ 1057us) server_negotiation.cc:407] Connection header received
0926 03:44:37.706943 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:44:37.706951 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:44:37.707041 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:44:37.707148 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:44:37.717036 (+ 9888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:37.718694 (+ 1658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:37.720892 (+ 2198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:37.721596 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:37.721710 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:37.726274 (+ 4564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:44:37.726281 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:44:37.726285 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:44:37.726324 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:44:37.733074 (+ 6750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:37.734418 (+ 1344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:37.734422 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:37.734424 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:37.734576 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:37.735037 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:37.735042 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:37.735044 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:37.735396 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:44:37.735647 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0926 03:44:37.736072 (+ 425us) server_negotiation.cc:299] Negotiation successful
0926 03:44:37.736285 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25986,"thread_start_us":26601,"threads_started":1}
I20260926 03:44:37.743315 8193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:37.655299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:44939 (local address 127.0.194.195:42269)
0926 03:44:37.666605 (+ 11306us) negotiation.cc:107] Waiting for socket to connect
0926 03:44:37.666681 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0926 03:44:37.669792 (+ 3111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:44:37.675932 (+ 6140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:44:37.675943 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:44:37.676492 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:44:37.677472 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:37.677496 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:37.678977 (+ 1481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:37.678984 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:37.729303 (+ 50319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:37.729322 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:37.730108 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:37.730116 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:37.730605 (+ 489us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:37.731291 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:44:37.731313 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:44:37.736861 (+ 5548us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:44:37.741004 (+ 4143us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:44:37.741008 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:44:37.741010 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:44:37.741356 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:44:37.741764 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:44:37.741767 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:44:37.741769 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:44:37.741895 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:44:37.742481 (+ 586us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:44:37.742484 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:44:37.742614 (+ 130us) client_negotiation.cc:769] Sending connection context
0926 03:44:37.742716 (+ 102us) client_negotiation.cc:240] Negotiation successful
0926 03:44:37.743005 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10883,"mutex_wait_us":2,"thread_start_us":14959,"threads_started":1}
I20260926 03:44:37.743983 8192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:37.654615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.195:42269 (local address 127.0.194.254:44939)
0926 03:44:37.675490 (+ 20875us) server_negotiation.cc:206] Beginning negotiation
0926 03:44:37.675500 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 03:44:37.675538 (+ 38us) server_negotiation.cc:407] Connection header received
0926 03:44:37.675616 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:44:37.675624 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:44:37.675724 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:44:37.675876 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:44:37.677687 (+ 1811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:37.678795 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:37.729449 (+ 50654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:37.729894 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:37.730046 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:37.737058 (+ 7012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:44:37.737063 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:44:37.737066 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:44:37.737095 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:44:37.740852 (+ 3757us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:37.741477 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:37.741484 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:37.741492 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:37.741658 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:37.742003 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:37.742008 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:37.742012 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:37.742360 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:44:37.742524 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0926 03:44:37.742897 (+ 373us) server_negotiation.cc:299] Negotiation successful
0926 03:44:37.743117 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20713,"thread_start_us":20592,"threads_started":1}
I20260926 03:44:37.744829 8189 heartbeater.cc:347] Connected to a master server at 127.0.194.254:44939
I20260926 03:44:37.744972 8189 heartbeater.cc:464] Registering TS with master...
I20260926 03:44:37.745409 8189 heartbeater.cc:511] Master 127.0.194.254:44939 requested a full tablet report, sending...
I20260926 03:44:37.746387 6932 ts_manager.cc:194] Re-registered known tserver with Master: 162b5f9a168f49db9ec3b1a5b0f0a526 (127.0.194.195:34367)
I20260926 03:44:37.747575 6932 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.194.195@KRBTEST.COM'} at 127.0.194.195:42269
I20260926 03:44:37.740731 6933 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57254:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 03:44:37.750659 6933 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 03:44:37.793283 8217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:37.768843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.193:32927 (local address 127.0.0.1:52164)
0926 03:44:37.780517 (+ 11674us) negotiation.cc:107] Waiting for socket to connect
0926 03:44:37.780555 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 03:44:37.780671 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:44:37.783479 (+ 2808us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:44:37.783483 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:44:37.783507 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:44:37.783861 (+ 354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:37.783872 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:37.786360 (+ 2488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:37.786365 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:37.788735 (+ 2370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:37.788755 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:37.791519 (+ 2764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:37.791524 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:37.792312 (+ 788us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:37.792370 (+ 58us) client_negotiation.cc:769] Sending connection context
0926 03:44:37.792468 (+ 98us) client_negotiation.cc:240] Negotiation successful
0926 03:44:37.792670 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11540,"mutex_wait_us":470,"thread_start_us":20889,"threads_started":1}
I20260926 03:44:37.794643 8216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:37.769042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52164 (local address 127.0.194.193:32927)
0926 03:44:37.782613 (+ 13571us) server_negotiation.cc:206] Beginning negotiation
0926 03:44:37.782624 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 03:44:37.782669 (+ 45us) server_negotiation.cc:407] Connection header received
0926 03:44:37.782795 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:44:37.782804 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:44:37.783069 (+ 265us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:44:37.783259 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:44:37.784100 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:37.786211 (+ 2111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:37.788928 (+ 2717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:37.791112 (+ 2184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:37.791848 (+ 736us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:37.792162 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0926 03:44:37.793979 (+ 1817us) server_negotiation.cc:299] Negotiation successful
0926 03:44:37.794226 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13401,"thread_start_us":14975,"threads_started":1}
I20260926 03:44:37.799609 7835 tablet_service.cc:1467] Processing CreateTablet for tablet 28298021129b453fafd36bfddde0496d (DEFAULT_TABLE table=test-table [id=510b25fb057a4d06a81f367cff72400a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:44:37.800647 8215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:37.769726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.194:42589 (local address 127.0.0.1:38660)
0926 03:44:37.790001 (+ 20275us) negotiation.cc:107] Waiting for socket to connect
0926 03:44:37.790019 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 03:44:37.790255 (+ 236us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:44:37.791515 (+ 1260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:44:37.791523 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:44:37.791558 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:44:37.792101 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:37.792115 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:37.795765 (+ 3650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:37.795771 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:37.798082 (+ 2311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:37.798096 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:37.799949 (+ 1853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:37.799957 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:37.800075 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:37.800097 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 03:44:37.800167 (+ 70us) client_negotiation.cc:240] Negotiation successful
0926 03:44:37.800250 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20110,"mutex_wait_us":1,"thread_start_us":10529,"threads_started":1}
I20260926 03:44:37.800765 8218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:37.770252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38660 (local address 127.0.194.194:42589)
0926 03:44:37.779707 (+ 9455us) server_negotiation.cc:206] Beginning negotiation
0926 03:44:37.779721 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 03:44:37.790139 (+ 10418us) server_negotiation.cc:407] Connection header received
0926 03:44:37.790506 (+ 367us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:44:37.790518 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:44:37.790876 (+ 358us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:44:37.791100 (+ 224us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:44:37.793920 (+ 2820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:37.795628 (+ 1708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:37.798263 (+ 2635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:37.799684 (+ 1421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:37.799864 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:37.800045 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0926 03:44:37.800236 (+ 191us) server_negotiation.cc:299] Negotiation successful
0926 03:44:37.800403 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9222,"thread_start_us":9111,"threads_started":1}
I20260926 03:44:37.801359 7835 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28298021129b453fafd36bfddde0496d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:44:37.803048 8217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:37.790165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.195:34367 (local address 127.0.0.1:41422)
0926 03:44:37.793564 (+ 3399us) negotiation.cc:107] Waiting for socket to connect
0926 03:44:37.793588 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 03:44:37.793701 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:44:37.796233 (+ 2532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:44:37.796238 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:44:37.796261 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:44:37.796645 (+ 384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:37.796653 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:37.798195 (+ 1542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:37.798203 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:37.799806 (+ 1603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:37.799834 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:37.802280 (+ 2446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:37.802286 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:37.802469 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:37.802501 (+ 32us) client_negotiation.cc:769] Sending connection context
0926 03:44:37.802608 (+ 107us) client_negotiation.cc:240] Negotiation successful
0926 03:44:37.802791 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3277,"thread_start_us":4847,"threads_started":1}
I20260926 03:44:37.803140 8220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:37.790432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41422 (local address 127.0.194.195:34367)
0926 03:44:37.795649 (+ 5217us) server_negotiation.cc:206] Beginning negotiation
0926 03:44:37.795658 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:44:37.795687 (+ 29us) server_negotiation.cc:407] Connection header received
0926 03:44:37.795776 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:44:37.795783 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:44:37.796017 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:44:37.796784 (+ 767us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:44:37.796997 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:37.798069 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:37.800866 (+ 2797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:37.802121 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:37.802265 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:37.802453 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0926 03:44:37.802662 (+ 209us) server_negotiation.cc:299] Negotiation successful
0926 03:44:37.802838 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5063,"thread_start_us":5706,"threads_started":1}
I20260926 03:44:37.804100 7982 tablet_service.cc:1467] Processing CreateTablet for tablet 28298021129b453fafd36bfddde0496d (DEFAULT_TABLE table=test-table [id=510b25fb057a4d06a81f367cff72400a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:44:37.805110 7982 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28298021129b453fafd36bfddde0496d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:44:37.808759 8123 tablet_service.cc:1467] Processing CreateTablet for tablet 28298021129b453fafd36bfddde0496d (DEFAULT_TABLE table=test-table [id=510b25fb057a4d06a81f367cff72400a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:44:37.810141 8123 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28298021129b453fafd36bfddde0496d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:44:37.814344 8221 tablet_bootstrap.cc:492] T 28298021129b453fafd36bfddde0496d P b08e95643f1a4377b8c2d42ac8762a70: Bootstrap starting.
I20260926 03:44:37.817487 8221 tablet_bootstrap.cc:654] T 28298021129b453fafd36bfddde0496d P b08e95643f1a4377b8c2d42ac8762a70: Neither blocks nor log segments found. Creating new log.
I20260926 03:44:37.818480 8221 log.cc:824] T 28298021129b453fafd36bfddde0496d P b08e95643f1a4377b8c2d42ac8762a70: Log is configured to *not* fsync() on all Append() calls
I20260926 03:44:37.821451 8221 tablet_bootstrap.cc:492] T 28298021129b453fafd36bfddde0496d P b08e95643f1a4377b8c2d42ac8762a70: No bootstrap required, opened a new log
I20260926 03:44:37.821722 8221 ts_tablet_manager.cc:1397] T 28298021129b453fafd36bfddde0496d P b08e95643f1a4377b8c2d42ac8762a70: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.003s
I20260926 03:44:37.823045 8223 tablet_bootstrap.cc:492] T 28298021129b453fafd36bfddde0496d P 949d66f722dd4b5c86409272002b87b3: Bootstrap starting.
I20260926 03:44:37.825873 8224 tablet_bootstrap.cc:492] T 28298021129b453fafd36bfddde0496d P 162b5f9a168f49db9ec3b1a5b0f0a526: Bootstrap starting.
I20260926 03:44:37.827509 8223 tablet_bootstrap.cc:654] T 28298021129b453fafd36bfddde0496d P 949d66f722dd4b5c86409272002b87b3: Neither blocks nor log segments found. Creating new log.
I20260926 03:44:37.828454 8223 log.cc:824] T 28298021129b453fafd36bfddde0496d P 949d66f722dd4b5c86409272002b87b3: Log is configured to *not* fsync() on all Append() calls
I20260926 03:44:37.829218 8224 tablet_bootstrap.cc:654] T 28298021129b453fafd36bfddde0496d P 162b5f9a168f49db9ec3b1a5b0f0a526: Neither blocks nor log segments found. Creating new log.
I20260926 03:44:37.830329 8224 log.cc:824] T 28298021129b453fafd36bfddde0496d P 162b5f9a168f49db9ec3b1a5b0f0a526: Log is configured to *not* fsync() on all Append() calls
I20260926 03:44:37.831061 8223 tablet_bootstrap.cc:492] T 28298021129b453fafd36bfddde0496d P 949d66f722dd4b5c86409272002b87b3: No bootstrap required, opened a new log
I20260926 03:44:37.831380 8223 ts_tablet_manager.cc:1397] T 28298021129b453fafd36bfddde0496d P 949d66f722dd4b5c86409272002b87b3: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260926 03:44:37.833619 8221 raft_consensus.cc:348] T 28298021129b453fafd36bfddde0496d P b08e95643f1a4377b8c2d42ac8762a70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b08e95643f1a4377b8c2d42ac8762a70" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 32927 } } peers { permanent_uuid: "949d66f722dd4b5c86409272002b87b3" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 42589 } } peers { permanent_uuid: "162b5f9a168f49db9ec3b1a5b0f0a526" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 34367 } }
I20260926 03:44:37.834065 8221 raft_consensus.cc:374] T 28298021129b453fafd36bfddde0496d P b08e95643f1a4377b8c2d42ac8762a70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:44:37.834173 8221 raft_consensus.cc:729] T 28298021129b453fafd36bfddde0496d P b08e95643f1a4377b8c2d42ac8762a70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b08e95643f1a4377b8c2d42ac8762a70, State: Initialized, Role: FOLLOWER
I20260926 03:44:37.834277 8224 tablet_bootstrap.cc:492] T 28298021129b453fafd36bfddde0496d P 162b5f9a168f49db9ec3b1a5b0f0a526: No bootstrap required, opened a new log
I20260926 03:44:37.834539 8224 ts_tablet_manager.cc:1397] T 28298021129b453fafd36bfddde0496d P 162b5f9a168f49db9ec3b1a5b0f0a526: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20260926 03:44:37.834997 8221 consensus_queue.cc:260] T 28298021129b453fafd36bfddde0496d P b08e95643f1a4377b8c2d42ac8762a70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b08e95643f1a4377b8c2d42ac8762a70" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 32927 } } peers { permanent_uuid: "949d66f722dd4b5c86409272002b87b3" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 42589 } } peers { permanent_uuid: "162b5f9a168f49db9ec3b1a5b0f0a526" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 34367 } }
I20260926 03:44:37.836751 8221 ts_tablet_manager.cc:1428] T 28298021129b453fafd36bfddde0496d P b08e95643f1a4377b8c2d42ac8762a70: Time spent starting tablet: real 0.015s user 0.013s sys 0.001s
I20260926 03:44:37.841373 8223 raft_consensus.cc:348] T 28298021129b453fafd36bfddde0496d P 949d66f722dd4b5c86409272002b87b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b08e95643f1a4377b8c2d42ac8762a70" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 32927 } } peers { permanent_uuid: "949d66f722dd4b5c86409272002b87b3" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 42589 } } peers { permanent_uuid: "162b5f9a168f49db9ec3b1a5b0f0a526" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 34367 } }
I20260926 03:44:37.841825 8223 raft_consensus.cc:374] T 28298021129b453fafd36bfddde0496d P 949d66f722dd4b5c86409272002b87b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:44:37.842109 8223 raft_consensus.cc:729] T 28298021129b453fafd36bfddde0496d P 949d66f722dd4b5c86409272002b87b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 949d66f722dd4b5c86409272002b87b3, State: Initialized, Role: FOLLOWER
I20260926 03:44:37.842772 8223 consensus_queue.cc:260] T 28298021129b453fafd36bfddde0496d P 949d66f722dd4b5c86409272002b87b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b08e95643f1a4377b8c2d42ac8762a70" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 32927 } } peers { permanent_uuid: "949d66f722dd4b5c86409272002b87b3" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 42589 } } peers { permanent_uuid: "162b5f9a168f49db9ec3b1a5b0f0a526" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 34367 } }
I20260926 03:44:37.844527 8223 ts_tablet_manager.cc:1428] T 28298021129b453fafd36bfddde0496d P 949d66f722dd4b5c86409272002b87b3: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260926 03:44:37.845167 8048 heartbeater.cc:503] Master 127.0.194.254:44939 was elected leader, sending a full tablet report...
I20260926 03:44:37.850322 8224 raft_consensus.cc:348] T 28298021129b453fafd36bfddde0496d P 162b5f9a168f49db9ec3b1a5b0f0a526 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b08e95643f1a4377b8c2d42ac8762a70" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 32927 } } peers { permanent_uuid: "949d66f722dd4b5c86409272002b87b3" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 42589 } } peers { permanent_uuid: "162b5f9a168f49db9ec3b1a5b0f0a526" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 34367 } }
I20260926 03:44:37.850689 8224 raft_consensus.cc:374] T 28298021129b453fafd36bfddde0496d P 162b5f9a168f49db9ec3b1a5b0f0a526 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:44:37.850808 8224 raft_consensus.cc:729] T 28298021129b453fafd36bfddde0496d P 162b5f9a168f49db9ec3b1a5b0f0a526 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 162b5f9a168f49db9ec3b1a5b0f0a526, State: Initialized, Role: FOLLOWER
I20260926 03:44:37.851361 8224 consensus_queue.cc:260] T 28298021129b453fafd36bfddde0496d P 162b5f9a168f49db9ec3b1a5b0f0a526 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b08e95643f1a4377b8c2d42ac8762a70" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 32927 } } peers { permanent_uuid: "949d66f722dd4b5c86409272002b87b3" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 42589 } } peers { permanent_uuid: "162b5f9a168f49db9ec3b1a5b0f0a526" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 34367 } }
I20260926 03:44:37.852386 8189 heartbeater.cc:503] Master 127.0.194.254:44939 was elected leader, sending a full tablet report...
I20260926 03:44:37.853034 8224 ts_tablet_manager.cc:1428] T 28298021129b453fafd36bfddde0496d P 162b5f9a168f49db9ec3b1a5b0f0a526: Time spent starting tablet: real 0.018s user 0.007s sys 0.007s
I20260926 03:44:37.867405 8228 raft_consensus.cc:482] T 28298021129b453fafd36bfddde0496d P 949d66f722dd4b5c86409272002b87b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 03:44:37.867652 8228 raft_consensus.cc:504] T 28298021129b453fafd36bfddde0496d P 949d66f722dd4b5c86409272002b87b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b08e95643f1a4377b8c2d42ac8762a70" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 32927 } } peers { permanent_uuid: "949d66f722dd4b5c86409272002b87b3" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 42589 } } peers { permanent_uuid: "162b5f9a168f49db9ec3b1a5b0f0a526" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 34367 } }
I20260926 03:44:37.869199 8228 leader_election.cc:288] T 28298021129b453fafd36bfddde0496d P 949d66f722dd4b5c86409272002b87b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b08e95643f1a4377b8c2d42ac8762a70 (127.0.194.193:32927), 162b5f9a168f49db9ec3b1a5b0f0a526 (127.0.194.195:34367)
I20260926 03:44:37.884009 8216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:37.869485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.194:52497 (local address 127.0.194.193:32927)
0926 03:44:37.869743 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0926 03:44:37.869753 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 03:44:37.876099 (+ 6346us) server_negotiation.cc:407] Connection header received
0926 03:44:37.876410 (+ 311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:44:37.876416 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:44:37.876506 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:44:37.876567 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:44:37.877361 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:37.878449 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:37.881668 (+ 3219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:37.882921 (+ 1253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:37.883082 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:37.883156 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 03:44:37.883646 (+ 490us) server_negotiation.cc:299] Negotiation successful
0926 03:44:37.883769 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 03:44:37.884009 8220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:37.875468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.194:50469 (local address 127.0.194.195:34367)
0926 03:44:37.875660 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0926 03:44:37.875664 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:44:37.876329 (+ 665us) server_negotiation.cc:407] Connection header received
0926 03:44:37.876447 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:44:37.876451 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:44:37.876538 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:44:37.876660 (+ 122us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:44:37.877358 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:37.878449 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:37.881668 (+ 3219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:37.882919 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:37.883082 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:37.883156 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 03:44:37.883646 (+ 490us) server_negotiation.cc:299] Negotiation successful
0926 03:44:37.883769 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260926 03:44:37.884439 8231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:37.875384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.195:34367 (local address 127.0.194.194:50469)
0926 03:44:37.876223 (+ 839us) negotiation.cc:107] Waiting for socket to connect
0926 03:44:37.876239 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 03:44:37.876342 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:44:37.876679 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:44:37.876687 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:44:37.876715 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:44:37.877196 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:37.877208 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:37.878599 (+ 1391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:37.878603 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:37.881509 (+ 2906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:37.881521 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:37.883150 (+ 1629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:37.883161 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:37.883432 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:37.883522 (+ 90us) client_negotiation.cc:769] Sending connection context
0926 03:44:37.883611 (+ 89us) client_negotiation.cc:240] Negotiation successful
0926 03:44:37.883811 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":739,"mutex_wait_us":415,"spinlock_wait_cycles":768,"thread_start_us":600,"threads_started":1}
I20260926 03:44:37.884649 8230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:37.869407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.193:32927 (local address 127.0.194.194:52497)
0926 03:44:37.875972 (+ 6565us) negotiation.cc:107] Waiting for socket to connect
0926 03:44:37.875996 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 03:44:37.876222 (+ 226us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:44:37.876751 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:44:37.876757 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:44:37.876780 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:44:37.877196 (+ 416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:37.877208 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:37.878598 (+ 1390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:37.878604 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:37.881508 (+ 2904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:37.881521 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:37.883170 (+ 1649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:37.883174 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:37.883435 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:37.883519 (+ 84us) client_negotiation.cc:769] Sending connection context
0926 03:44:37.883611 (+ 92us) client_negotiation.cc:240] Negotiation successful
0926 03:44:37.883811 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6388,"mutex_wait_us":344,"thread_start_us":5509,"threads_started":1}
I20260926 03:44:37.885479 7855 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28298021129b453fafd36bfddde0496d" candidate_uuid: "949d66f722dd4b5c86409272002b87b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b08e95643f1a4377b8c2d42ac8762a70" is_pre_election: true
I20260926 03:44:37.885470 8143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28298021129b453fafd36bfddde0496d" candidate_uuid: "949d66f722dd4b5c86409272002b87b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "162b5f9a168f49db9ec3b1a5b0f0a526" is_pre_election: true
I20260926 03:44:37.885942 8143 raft_consensus.cc:2454] T 28298021129b453fafd36bfddde0496d P 162b5f9a168f49db9ec3b1a5b0f0a526 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 949d66f722dd4b5c86409272002b87b3 in term 0.
I20260926 03:44:37.885949 7855 raft_consensus.cc:2454] T 28298021129b453fafd36bfddde0496d P b08e95643f1a4377b8c2d42ac8762a70 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 949d66f722dd4b5c86409272002b87b3 in term 0.
I20260926 03:44:37.886559 7935 leader_election.cc:302] T 28298021129b453fafd36bfddde0496d P 949d66f722dd4b5c86409272002b87b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 162b5f9a168f49db9ec3b1a5b0f0a526, 949d66f722dd4b5c86409272002b87b3; no voters:
I20260926 03:44:37.886942 8228 raft_consensus.cc:2781] T 28298021129b453fafd36bfddde0496d P 949d66f722dd4b5c86409272002b87b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 03:44:37.887097 8228 raft_consensus.cc:482] T 28298021129b453fafd36bfddde0496d P 949d66f722dd4b5c86409272002b87b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 03:44:37.887225 8228 raft_consensus.cc:3037] T 28298021129b453fafd36bfddde0496d P 949d66f722dd4b5c86409272002b87b3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:44:37.889392 8228 raft_consensus.cc:504] T 28298021129b453fafd36bfddde0496d P 949d66f722dd4b5c86409272002b87b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b08e95643f1a4377b8c2d42ac8762a70" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 32927 } } peers { permanent_uuid: "949d66f722dd4b5c86409272002b87b3" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 42589 } } peers { permanent_uuid: "162b5f9a168f49db9ec3b1a5b0f0a526" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 34367 } }
I20260926 03:44:37.890384 8228 leader_election.cc:288] T 28298021129b453fafd36bfddde0496d P 949d66f722dd4b5c86409272002b87b3 [CANDIDATE]: Term 1 election: Requested vote from peers b08e95643f1a4377b8c2d42ac8762a70 (127.0.194.193:32927), 162b5f9a168f49db9ec3b1a5b0f0a526 (127.0.194.195:34367)
I20260926 03:44:37.890816 7855 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28298021129b453fafd36bfddde0496d" candidate_uuid: "949d66f722dd4b5c86409272002b87b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b08e95643f1a4377b8c2d42ac8762a70"
I20260926 03:44:37.890885 8143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28298021129b453fafd36bfddde0496d" candidate_uuid: "949d66f722dd4b5c86409272002b87b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "162b5f9a168f49db9ec3b1a5b0f0a526"
I20260926 03:44:37.891006 7855 raft_consensus.cc:3037] T 28298021129b453fafd36bfddde0496d P b08e95643f1a4377b8c2d42ac8762a70 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:44:37.891084 8143 raft_consensus.cc:3037] T 28298021129b453fafd36bfddde0496d P 162b5f9a168f49db9ec3b1a5b0f0a526 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:44:37.893144 7855 raft_consensus.cc:2454] T 28298021129b453fafd36bfddde0496d P b08e95643f1a4377b8c2d42ac8762a70 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 949d66f722dd4b5c86409272002b87b3 in term 1.
I20260926 03:44:37.893150 8143 raft_consensus.cc:2454] T 28298021129b453fafd36bfddde0496d P 162b5f9a168f49db9ec3b1a5b0f0a526 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 949d66f722dd4b5c86409272002b87b3 in term 1.
I20260926 03:44:37.893488 7935 leader_election.cc:302] T 28298021129b453fafd36bfddde0496d P 949d66f722dd4b5c86409272002b87b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 949d66f722dd4b5c86409272002b87b3, b08e95643f1a4377b8c2d42ac8762a70; no voters:
I20260926 03:44:37.893790 8228 raft_consensus.cc:2781] T 28298021129b453fafd36bfddde0496d P 949d66f722dd4b5c86409272002b87b3 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:44:37.894220 8228 raft_consensus.cc:686] T 28298021129b453fafd36bfddde0496d P 949d66f722dd4b5c86409272002b87b3 [term 1 LEADER]: Becoming Leader. State: Replica: 949d66f722dd4b5c86409272002b87b3, State: Running, Role: LEADER
I20260926 03:44:37.894788 8228 consensus_queue.cc:237] T 28298021129b453fafd36bfddde0496d P 949d66f722dd4b5c86409272002b87b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b08e95643f1a4377b8c2d42ac8762a70" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 32927 } } peers { permanent_uuid: "949d66f722dd4b5c86409272002b87b3" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 42589 } } peers { permanent_uuid: "162b5f9a168f49db9ec3b1a5b0f0a526" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 34367 } }
I20260926 03:44:37.900763 6932 catalog_manager.cc:5495] T 28298021129b453fafd36bfddde0496d P 949d66f722dd4b5c86409272002b87b3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 949d66f722dd4b5c86409272002b87b3 (127.0.194.194). New cstate: current_term: 1 leader_uuid: "949d66f722dd4b5c86409272002b87b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b08e95643f1a4377b8c2d42ac8762a70" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 32927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "949d66f722dd4b5c86409272002b87b3" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 42589 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "162b5f9a168f49db9ec3b1a5b0f0a526" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 34367 } health_report { overall_health: UNKNOWN } } }
W20260926 03:44:37.903890 8191 tablet.cc:2369] T 28298021129b453fafd36bfddde0496d P 162b5f9a168f49db9ec3b1a5b0f0a526: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 03:44:37.942456 8218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:37.937079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38672 (local address 127.0.194.194:42589)
0926 03:44:37.937281 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0926 03:44:37.937286 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:44:37.937304 (+ 18us) server_negotiation.cc:407] Connection header received
0926 03:44:37.937518 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:44:37.937522 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:44:37.937597 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:44:37.937750 (+ 153us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:44:37.938107 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:37.938912 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:37.940389 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:37.940784 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:37.940918 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:37.941395 (+ 477us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:44:37.941886 (+ 491us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:44:37.942063 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0926 03:44:37.942109 (+ 46us) server_negotiation.cc:299] Negotiation successful
0926 03:44:37.942187 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
W20260926 03:44:37.947669 7903 tablet.cc:2369] T 28298021129b453fafd36bfddde0496d P b08e95643f1a4377b8c2d42ac8762a70: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 03:44:37.950306 8143 raft_consensus.cc:1261] T 28298021129b453fafd36bfddde0496d P 162b5f9a168f49db9ec3b1a5b0f0a526 [term 1 FOLLOWER]: Refusing update from remote peer 949d66f722dd4b5c86409272002b87b3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:44:37.950325 7855 raft_consensus.cc:1261] T 28298021129b453fafd36bfddde0496d P b08e95643f1a4377b8c2d42ac8762a70 [term 1 FOLLOWER]: Refusing update from remote peer 949d66f722dd4b5c86409272002b87b3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:44:37.951211 8232 consensus_queue.cc:1035] T 28298021129b453fafd36bfddde0496d P 949d66f722dd4b5c86409272002b87b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b08e95643f1a4377b8c2d42ac8762a70" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:44:37.951658 8228 consensus_queue.cc:1035] T 28298021129b453fafd36bfddde0496d P 949d66f722dd4b5c86409272002b87b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "162b5f9a168f49db9ec3b1a5b0f0a526" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 34367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:44:37.964140 8236 mvcc.cc:205] Tried to move back new op lower bound from 7333454962475515904 to 7333454962261098496. Current Snapshot: MvccSnapshot[applied={T|T < 7333454962475515904}]
I20260926 03:44:37.980120 8237 mvcc.cc:205] Tried to move back new op lower bound from 7333454962475515904 to 7333454962261098496. Current Snapshot: MvccSnapshot[applied={T|T < 7333454962475515904}]
I20260926 03:44:37.981014 6932 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57254:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:44:37.981283 6932 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57254:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:44:37.995836 6932 catalog_manager.cc:5782] T 00000000000000000000000000000000 P be6fefc6a90b41218affcf0df9459735: Sending DeleteTablet for 3 replicas of tablet 28298021129b453fafd36bfddde0496d
I20260926 03:44:37.997112 8123 tablet_service.cc:1514] Processing DeleteTablet for tablet 28298021129b453fafd36bfddde0496d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:44:37 UTC) from {username='oryx', principal='oryx/127.0.194.254@KRBTEST.COM'} at 127.0.0.1:41422
I20260926 03:44:37.997112 7835 tablet_service.cc:1514] Processing DeleteTablet for tablet 28298021129b453fafd36bfddde0496d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:44:37 UTC) from {username='oryx', principal='oryx/127.0.194.254@KRBTEST.COM'} at 127.0.0.1:52164
I20260926 03:44:37.997891 7982 tablet_service.cc:1514] Processing DeleteTablet for tablet 28298021129b453fafd36bfddde0496d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:44:37 UTC) from {username='oryx', principal='oryx/127.0.194.254@KRBTEST.COM'} at 127.0.0.1:38660
I20260926 03:44:37.998844 8248 tablet_replica.cc:330] T 28298021129b453fafd36bfddde0496d P 949d66f722dd4b5c86409272002b87b3: stopping tablet replica
I20260926 03:44:37.998994 8122 tablet_service.cc:1514] Processing DeleteTablet for tablet 28298021129b453fafd36bfddde0496d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:44:37 UTC) from {username='oryx', principal='oryx/127.0.194.254@KRBTEST.COM'} at 127.0.0.1:41422
I20260926 03:44:37.999298 8248 raft_consensus.cc:2229] T 28298021129b453fafd36bfddde0496d P 949d66f722dd4b5c86409272002b87b3 [term 1 LEADER]: Raft consensus shutting down.
I20260926 03:44:38.000295 8248 raft_consensus.cc:2258] T 28298021129b453fafd36bfddde0496d P 949d66f722dd4b5c86409272002b87b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:44:38.004482 779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu pid 6901
I20260926 03:44:38.005625 6963 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:44:38.015717 8248 ts_tablet_manager.cc:1905] T 28298021129b453fafd36bfddde0496d P 949d66f722dd4b5c86409272002b87b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:44:38.022308 8206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:38.002475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43934 (local address 127.0.194.253:37485)
0926 03:44:38.002684 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0926 03:44:38.002693 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:44:38.002718 (+ 25us) server_negotiation.cc:407] Connection header received
0926 03:44:38.002853 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:44:38.002858 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:44:38.002938 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:44:38.003004 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:44:38.005249 (+ 2245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:38.006369 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:38.008344 (+ 1975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:38.008999 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:38.009088 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:38.013840 (+ 4752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:44:38.013848 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:44:38.013854 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:44:38.013934 (+ 80us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:44:38.017828 (+ 3894us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:38.018835 (+ 1007us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:38.018841 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:38.018846 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:38.019005 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:38.019650 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:38.019658 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:38.019664 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:38.020185 (+ 521us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:44:38.020796 (+ 611us) server_negotiation.cc:1035] Waiting for connection context
0926 03:44:38.021689 (+ 893us) server_negotiation.cc:299] Negotiation successful
0926 03:44:38.021972 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 03:44:38.022297 8247 tablet_replica.cc:330] T 28298021129b453fafd36bfddde0496d P b08e95643f1a4377b8c2d42ac8762a70: stopping tablet replica
I20260926 03:44:38.022909 8247 raft_consensus.cc:2229] T 28298021129b453fafd36bfddde0496d P b08e95643f1a4377b8c2d42ac8762a70 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:44:38.023470 8247 raft_consensus.cc:2258] T 28298021129b453fafd36bfddde0496d P b08e95643f1a4377b8c2d42ac8762a70 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:44:38.028399 8248 ts_tablet_manager.cc:1918] T 28298021129b453fafd36bfddde0496d P 949d66f722dd4b5c86409272002b87b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:44:38.028576 8248 log.cc:1196] T 28298021129b453fafd36bfddde0496d P 949d66f722dd4b5c86409272002b87b3: Deleting WAL directory at /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/minicluster-data/ts-1/wal/wals/28298021129b453fafd36bfddde0496d
I20260926 03:44:38.029078 8247 ts_tablet_manager.cc:1905] T 28298021129b453fafd36bfddde0496d P b08e95643f1a4377b8c2d42ac8762a70: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:44:38.029186 8248 ts_tablet_manager.cc:1939] T 28298021129b453fafd36bfddde0496d P 949d66f722dd4b5c86409272002b87b3: Deleting consensus metadata
I20260926 03:44:38.031390 8246 tablet_replica.cc:330] T 28298021129b453fafd36bfddde0496d P 162b5f9a168f49db9ec3b1a5b0f0a526: stopping tablet replica
I20260926 03:44:38.032272 8246 raft_consensus.cc:2229] T 28298021129b453fafd36bfddde0496d P 162b5f9a168f49db9ec3b1a5b0f0a526 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:44:38.032749 8246 raft_consensus.cc:2258] T 28298021129b453fafd36bfddde0496d P 162b5f9a168f49db9ec3b1a5b0f0a526 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:44:38.034832 8246 ts_tablet_manager.cc:1905] T 28298021129b453fafd36bfddde0496d P 162b5f9a168f49db9ec3b1a5b0f0a526: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:44:38.036877 8247 ts_tablet_manager.cc:1918] T 28298021129b453fafd36bfddde0496d P b08e95643f1a4377b8c2d42ac8762a70: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:44:38.037029 8247 log.cc:1196] T 28298021129b453fafd36bfddde0496d P b08e95643f1a4377b8c2d42ac8762a70: Deleting WAL directory at /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/minicluster-data/ts-0/wal/wals/28298021129b453fafd36bfddde0496d
I20260926 03:44:38.037539 8247 ts_tablet_manager.cc:1939] T 28298021129b453fafd36bfddde0496d P b08e95643f1a4377b8c2d42ac8762a70: Deleting consensus metadata
I20260926 03:44:38.042485 8246 ts_tablet_manager.cc:1918] T 28298021129b453fafd36bfddde0496d P 162b5f9a168f49db9ec3b1a5b0f0a526: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:44:38.042651 8246 log.cc:1196] T 28298021129b453fafd36bfddde0496d P 162b5f9a168f49db9ec3b1a5b0f0a526: Deleting WAL directory at /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/minicluster-data/ts-2/wal/wals/28298021129b453fafd36bfddde0496d
I20260926 03:44:38.043215 8246 ts_tablet_manager.cc:1939] T 28298021129b453fafd36bfddde0496d P 162b5f9a168f49db9ec3b1a5b0f0a526: Deleting consensus metadata
I20260926 03:44:38.043488 8205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:38.003742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53038 (local address 127.0.194.252:34225)
0926 03:44:38.003958 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0926 03:44:38.003964 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:44:38.023267 (+ 19303us) server_negotiation.cc:407] Connection header received
0926 03:44:38.023571 (+ 304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:44:38.023576 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:44:38.023668 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:44:38.023785 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:44:38.025768 (+ 1983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:38.026948 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:38.029562 (+ 2614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:38.030374 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:38.030500 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:38.035512 (+ 5012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:44:38.035519 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:44:38.035523 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:44:38.035563 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:44:38.039729 (+ 4166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:38.040855 (+ 1126us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:38.040861 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:38.040864 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:38.041014 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:38.041493 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:38.041498 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:38.041502 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:38.041870 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:44:38.042087 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0926 03:44:38.042857 (+ 770us) server_negotiation.cc:299] Negotiation successful
0926 03:44:38.043090 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 03:44:38.172744 6917 catalog_manager.cc:4841] TS 162b5f9a168f49db9ec3b1a5b0f0a526 (127.0.194.195:34367): tablet 28298021129b453fafd36bfddde0496d (table test-table [id=510b25fb057a4d06a81f367cff72400a]) successfully deleted
I20260926 03:44:38.173017 6918 catalog_manager.cc:4841] TS 949d66f722dd4b5c86409272002b87b3 (127.0.194.194:42589): tablet 28298021129b453fafd36bfddde0496d (table test-table [id=510b25fb057a4d06a81f367cff72400a]) successfully deleted
W20260926 03:44:38.173218 6918 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.194.194:42589, user_credentials={real_user=slave}} blocked reactor thread for 142284us
W20260926 03:44:38.173337 6917 catalog_manager.cc:4808] TS 162b5f9a168f49db9ec3b1a5b0f0a526 (127.0.194.195:34367): delete failed for tablet 28298021129b453fafd36bfddde0496d because the tablet was not found. No further retry: Not found: Tablet not found: 28298021129b453fafd36bfddde0496d
I20260926 03:44:38.173605 6917 catalog_manager.cc:4841] TS b08e95643f1a4377b8c2d42ac8762a70 (127.0.194.193:32927): tablet 28298021129b453fafd36bfddde0496d (table test-table [id=510b25fb057a4d06a81f367cff72400a]) successfully deleted
I20260926 03:44:38.174177 779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu with pid 6901
I20260926 03:44:38.227766 8205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:38.208609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53046 (local address 127.0.194.252:34225)
0926 03:44:38.208800 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0926 03:44:38.208806 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:44:38.210699 (+ 1893us) server_negotiation.cc:407] Connection header received
0926 03:44:38.211031 (+ 332us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:44:38.211037 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:44:38.211129 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:44:38.211193 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:44:38.213989 (+ 2796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:38.215171 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:38.216971 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:38.217368 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:38.217510 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:38.220478 (+ 2968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:44:38.220484 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:44:38.220488 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:44:38.220532 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:44:38.223963 (+ 3431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:38.225324 (+ 1361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:38.225329 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:38.225333 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:38.225486 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:38.225948 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:38.225954 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:38.225958 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:38.226311 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:44:38.226556 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0926 03:44:38.226984 (+ 428us) server_negotiation.cc:299] Negotiation successful
0926 03:44:38.227204 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 03:44:38.228533 8206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:38.208626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43936 (local address 127.0.194.253:37485)
0926 03:44:38.208850 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0926 03:44:38.208859 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:44:38.210639 (+ 1780us) server_negotiation.cc:407] Connection header received
0926 03:44:38.211039 (+ 400us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:44:38.211045 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:44:38.211119 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:44:38.211193 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:44:38.214138 (+ 2945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:38.214950 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:38.216933 (+ 1983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:38.217563 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:38.217713 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:38.221332 (+ 3619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:44:38.221339 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:44:38.221344 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:44:38.221386 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:44:38.224471 (+ 3085us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:38.225342 (+ 871us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:38.225353 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:38.225356 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:38.225486 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:38.226157 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:38.226162 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:38.226166 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:38.226934 (+ 768us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:44:38.227165 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0926 03:44:38.227668 (+ 503us) server_negotiation.cc:299] Negotiation successful
0926 03:44:38.228214 (+ 546us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 03:44:38.974220 8258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:38.973454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:44939 (local address 127.0.194.193:60161)
0926 03:44:38.973957 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0926 03:44:38.974065 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.254:44939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":261,"threads_started":1}
W20260926 03:44:38.974622 7901 heartbeater.cc:644] Failed to heartbeat to 127.0.194.254:44939 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.194.254:44939: connect: Connection refused (error 111)
I20260926 03:44:38.981262 8259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:38.980446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:44939 (local address 127.0.194.194:38785)
0926 03:44:38.980918 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0926 03:44:38.981048 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.254:44939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":230,"threads_started":1}
W20260926 03:44:38.981629 8048 heartbeater.cc:644] Failed to heartbeat to 127.0.194.254:44939 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.194.254:44939: connect: Connection refused (error 111)
I20260926 03:44:38.998564 8260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:38.997810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:44939 (local address 127.0.194.195:33777)
0926 03:44:38.998265 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0926 03:44:38.998379 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.254:44939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":251,"threads_started":1}
W20260926 03:44:38.998958 8189 heartbeater.cc:644] Failed to heartbeat to 127.0.194.254:44939 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.194.254:44939: connect: Connection refused (error 111)
I20260926 03:44:39.493949 8261 raft_consensus.cc:482] T 00000000000000000000000000000000 P af7004004c624669ae81014cf255d63f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader be6fefc6a90b41218affcf0df9459735)
I20260926 03:44:39.494266 8261 raft_consensus.cc:504] T 00000000000000000000000000000000 P af7004004c624669ae81014cf255d63f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "be6fefc6a90b41218affcf0df9459735" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 44939 } } peers { permanent_uuid: "af7004004c624669ae81014cf255d63f" member_type: VOTER last_known_addr { host: "127.0.194.253" port: 37485 } attrs { promote: false } } peers { permanent_uuid: "037743be8b9845979457b84234b6bdad" member_type: VOTER last_known_addr { host: "127.0.194.252" port: 34225 } attrs { promote: false } }
I20260926 03:44:39.494972 8262 raft_consensus.cc:482] T 00000000000000000000000000000000 P 037743be8b9845979457b84234b6bdad [term 1 FOLLOWER]: Starting pre-election (detected failure of leader be6fefc6a90b41218affcf0df9459735)
I20260926 03:44:39.495216 8262 raft_consensus.cc:504] T 00000000000000000000000000000000 P 037743be8b9845979457b84234b6bdad [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "be6fefc6a90b41218affcf0df9459735" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 44939 } } peers { permanent_uuid: "af7004004c624669ae81014cf255d63f" member_type: VOTER last_known_addr { host: "127.0.194.253" port: 37485 } attrs { promote: false } } peers { permanent_uuid: "037743be8b9845979457b84234b6bdad" member_type: VOTER last_known_addr { host: "127.0.194.252" port: 34225 } attrs { promote: false } }
I20260926 03:44:39.495889 8261 leader_election.cc:288] T 00000000000000000000000000000000 P af7004004c624669ae81014cf255d63f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers be6fefc6a90b41218affcf0df9459735 (127.0.194.254:44939), 037743be8b9845979457b84234b6bdad (127.0.194.252:34225)
I20260926 03:44:39.496626 8262 leader_election.cc:288] T 00000000000000000000000000000000 P 037743be8b9845979457b84234b6bdad [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers be6fefc6a90b41218affcf0df9459735 (127.0.194.254:44939), af7004004c624669ae81014cf255d63f (127.0.194.253:37485)
I20260926 03:44:39.497758 8264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:39.495986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:44939 (local address 127.0.0.1:57678)
0926 03:44:39.497214 (+ 1228us) negotiation.cc:107] Waiting for socket to connect
0926 03:44:39.497404 (+ 190us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.254:44939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1098,"threads_started":1}
W20260926 03:44:39.500042 7518 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.194.254:44939: connect: Connection refused (error 111)
I20260926 03:44:39.507639 8264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:39.496130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.252:34225 (local address 127.0.0.1:53052)
0926 03:44:39.498002 (+ 1872us) negotiation.cc:107] Waiting for socket to connect
0926 03:44:39.498025 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 03:44:39.498195 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:44:39.500497 (+ 2302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:44:39.500502 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:44:39.500529 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:44:39.500783 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:39.500795 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:39.502506 (+ 1711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:39.502510 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:39.504634 (+ 2124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:39.504649 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:39.506871 (+ 2222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:39.506879 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:39.507059 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:39.507100 (+ 41us) client_negotiation.cc:769] Sending connection context
0926 03:44:39.507169 (+ 69us) client_negotiation.cc:240] Negotiation successful
0926 03:44:39.507314 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1736,"thread_start_us":823,"threads_started":1}
I20260926 03:44:39.507747 8266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:39.496782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.253:37485 (local address 127.0.0.1:43952)
0926 03:44:39.498967 (+ 2185us) negotiation.cc:107] Waiting for socket to connect
0926 03:44:39.498988 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 03:44:39.499134 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:44:39.500117 (+ 983us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:44:39.500121 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:44:39.500176 (+ 55us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:44:39.500529 (+ 353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:39.500540 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:39.502505 (+ 1965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:39.502510 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:39.504634 (+ 2124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:39.504650 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:39.506995 (+ 2345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:39.507001 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:39.507170 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:39.507203 (+ 33us) client_negotiation.cc:769] Sending connection context
0926 03:44:39.507260 (+ 57us) client_negotiation.cc:240] Negotiation successful
0926 03:44:39.507380 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1879,"thread_start_us":1706,"threads_started":1}
I20260926 03:44:39.508275 8266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:39.500190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:44939 (local address 127.0.0.1:57690)
0926 03:44:39.507985 (+ 7795us) negotiation.cc:107] Waiting for socket to connect
0926 03:44:39.508124 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.254:44939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":7633,"threads_started":1}
I20260926 03:44:39.508605 8265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:39.496542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53052 (local address 127.0.194.252:34225)
0926 03:44:39.500161 (+ 3619us) server_negotiation.cc:206] Beginning negotiation
0926 03:44:39.500166 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:44:39.500180 (+ 14us) server_negotiation.cc:407] Connection header received
0926 03:44:39.500225 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:44:39.500235 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:44:39.500322 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:44:39.500927 (+ 605us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:44:39.501118 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:39.502348 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:39.504832 (+ 2484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:39.506638 (+ 1806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:39.507995 (+ 1357us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:39.508111 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0926 03:44:39.508158 (+ 47us) server_negotiation.cc:299] Negotiation successful
0926 03:44:39.508282 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3498,"thread_start_us":3359,"threads_started":1}
I20260926 03:44:39.509248 8267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:39.497216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43952 (local address 127.0.194.253:37485)
0926 03:44:39.499730 (+ 2514us) server_negotiation.cc:206] Beginning negotiation
0926 03:44:39.499738 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:44:39.499762 (+ 24us) server_negotiation.cc:407] Connection header received
0926 03:44:39.499835 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:44:39.499843 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:44:39.499947 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:44:39.500675 (+ 728us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:44:39.500975 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:39.502345 (+ 1370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:39.505489 (+ 3144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:39.506768 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:39.508498 (+ 1730us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:39.508611 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0926 03:44:39.508662 (+ 51us) server_negotiation.cc:299] Negotiation successful
0926 03:44:39.508815 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2370,"thread_start_us":2190,"threads_started":1}
W20260926 03:44:39.509668 7706 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.194.254:44939: connect: Connection refused (error 111)
I20260926 03:44:39.509863 7540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "037743be8b9845979457b84234b6bdad" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "af7004004c624669ae81014cf255d63f" is_pre_election: true
I20260926 03:44:39.510121 8264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:39.509711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:44939 (local address 127.0.0.1:57720)
0926 03:44:39.509890 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0926 03:44:39.509976 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.254:44939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260926 03:44:39.510232 7540 raft_consensus.cc:2454] T 00000000000000000000000000000000 P af7004004c624669ae81014cf255d63f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 037743be8b9845979457b84234b6bdad in term 1.
W20260926 03:44:39.510385 7518 leader_election.cc:334] T 00000000000000000000000000000000 P af7004004c624669ae81014cf255d63f [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer be6fefc6a90b41218affcf0df9459735 (127.0.194.254:44939): Network error: Client connection negotiation failed: client connection to 127.0.194.254:44939: connect: Connection refused (error 111)
I20260926 03:44:39.510841 7705 leader_election.cc:302] T 00000000000000000000000000000000 P 037743be8b9845979457b84234b6bdad [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 037743be8b9845979457b84234b6bdad, af7004004c624669ae81014cf255d63f; no voters:
I20260926 03:44:39.511241 8262 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 037743be8b9845979457b84234b6bdad [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260926 03:44:39.511379 8262 raft_consensus.cc:482] T 00000000000000000000000000000000 P 037743be8b9845979457b84234b6bdad [term 1 FOLLOWER]: Starting leader election (detected failure of leader be6fefc6a90b41218affcf0df9459735)
I20260926 03:44:39.511528 8262 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 037743be8b9845979457b84234b6bdad [term 1 FOLLOWER]: Advancing to term 2
I20260926 03:44:39.511509 7730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "af7004004c624669ae81014cf255d63f" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "037743be8b9845979457b84234b6bdad" is_pre_election: true
I20260926 03:44:39.512562 8268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:39.512028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:44939 (local address 127.0.0.1:57724)
0926 03:44:39.512219 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0926 03:44:39.512365 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.254:44939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260926 03:44:39.512738 7706 leader_election.cc:334] T 00000000000000000000000000000000 P 037743be8b9845979457b84234b6bdad [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer be6fefc6a90b41218affcf0df9459735 (127.0.194.254:44939): Network error: Client connection negotiation failed: client connection to 127.0.194.254:44939: connect: Connection refused (error 111)
I20260926 03:44:39.513976 8262 raft_consensus.cc:504] T 00000000000000000000000000000000 P 037743be8b9845979457b84234b6bdad [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "be6fefc6a90b41218affcf0df9459735" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 44939 } } peers { permanent_uuid: "af7004004c624669ae81014cf255d63f" member_type: VOTER last_known_addr { host: "127.0.194.253" port: 37485 } attrs { promote: false } } peers { permanent_uuid: "037743be8b9845979457b84234b6bdad" member_type: VOTER last_known_addr { host: "127.0.194.252" port: 34225 } attrs { promote: false } }
I20260926 03:44:39.514385 7730 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 037743be8b9845979457b84234b6bdad [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af7004004c624669ae81014cf255d63f in current term 2: Already voted for candidate 037743be8b9845979457b84234b6bdad in this term.
I20260926 03:44:39.514629 8262 leader_election.cc:288] T 00000000000000000000000000000000 P 037743be8b9845979457b84234b6bdad [CANDIDATE]: Term 2 election: Requested vote from peers be6fefc6a90b41218affcf0df9459735 (127.0.194.254:44939), af7004004c624669ae81014cf255d63f (127.0.194.253:37485)
I20260926 03:44:39.515130 8268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:39.514787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:44939 (local address 127.0.0.1:57734)
0926 03:44:39.514923 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0926 03:44:39.515009 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.254:44939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260926 03:44:39.515123 7540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "037743be8b9845979457b84234b6bdad" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "af7004004c624669ae81014cf255d63f"
I20260926 03:44:39.515350 7540 raft_consensus.cc:3037] T 00000000000000000000000000000000 P af7004004c624669ae81014cf255d63f [term 1 FOLLOWER]: Advancing to term 2
I20260926 03:44:39.515882 7516 leader_election.cc:302] T 00000000000000000000000000000000 P af7004004c624669ae81014cf255d63f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af7004004c624669ae81014cf255d63f; no voters: 037743be8b9845979457b84234b6bdad, be6fefc6a90b41218affcf0df9459735
I20260926 03:44:39.516458 8268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:39.516099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:44939 (local address 127.0.0.1:57742)
0926 03:44:39.516234 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0926 03:44:39.516311 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.254:44939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260926 03:44:39.516670 7706 leader_election.cc:334] T 00000000000000000000000000000000 P 037743be8b9845979457b84234b6bdad [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer be6fefc6a90b41218affcf0df9459735 (127.0.194.254:44939): Network error: Client connection negotiation failed: client connection to 127.0.194.254:44939: connect: Connection refused (error 111)
I20260926 03:44:39.517850 7540 raft_consensus.cc:2454] T 00000000000000000000000000000000 P af7004004c624669ae81014cf255d63f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 037743be8b9845979457b84234b6bdad in term 2.
I20260926 03:44:39.518019 8261 raft_consensus.cc:2726] T 00000000000000000000000000000000 P af7004004c624669ae81014cf255d63f [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 03:44:39.518347 7705 leader_election.cc:302] T 00000000000000000000000000000000 P 037743be8b9845979457b84234b6bdad [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 037743be8b9845979457b84234b6bdad, af7004004c624669ae81014cf255d63f; no voters: be6fefc6a90b41218affcf0df9459735
I20260926 03:44:39.518698 8262 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 037743be8b9845979457b84234b6bdad [term 2 FOLLOWER]: Leader election won for term 2
I20260926 03:44:39.519148 8262 raft_consensus.cc:686] T 00000000000000000000000000000000 P 037743be8b9845979457b84234b6bdad [term 2 LEADER]: Becoming Leader. State: Replica: 037743be8b9845979457b84234b6bdad, State: Running, Role: LEADER
I20260926 03:44:39.519513 8262 consensus_queue.cc:237] T 00000000000000000000000000000000 P 037743be8b9845979457b84234b6bdad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "be6fefc6a90b41218affcf0df9459735" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 44939 } } peers { permanent_uuid: "af7004004c624669ae81014cf255d63f" member_type: VOTER last_known_addr { host: "127.0.194.253" port: 37485 } attrs { promote: false } } peers { permanent_uuid: "037743be8b9845979457b84234b6bdad" member_type: VOTER last_known_addr { host: "127.0.194.252" port: 34225 } attrs { promote: false } }
I20260926 03:44:39.521342 8271 sys_catalog.cc:455] T 00000000000000000000000000000000 P 037743be8b9845979457b84234b6bdad [sys.catalog]: SysCatalogTable state changed. Reason: New leader 037743be8b9845979457b84234b6bdad. Latest consensus state: current_term: 2 leader_uuid: "037743be8b9845979457b84234b6bdad" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "be6fefc6a90b41218affcf0df9459735" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 44939 } } peers { permanent_uuid: "af7004004c624669ae81014cf255d63f" member_type: VOTER last_known_addr { host: "127.0.194.253" port: 37485 } attrs { promote: false } } peers { permanent_uuid: "037743be8b9845979457b84234b6bdad" member_type: VOTER last_known_addr { host: "127.0.194.252" port: 34225 } attrs { promote: false } } }
I20260926 03:44:39.521696 8271 sys_catalog.cc:458] T 00000000000000000000000000000000 P 037743be8b9845979457b84234b6bdad [sys.catalog]: This master's current role is: LEADER
I20260926 03:44:39.522249 8273 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:44:39.525753 8273 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:44:39.526530 8273 catalog_manager.cc:1259] Loaded cluster ID: 5900f64ad7ae4b009c7ea641aa872bad
I20260926 03:44:39.526598 8273 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:44:39.527777 8273 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:44:39.528556 8273 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 037743be8b9845979457b84234b6bdad: Loaded TSK: 0
I20260926 03:44:39.529376 8273 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 03:44:39.567039 7720 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53046:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 03:44:39.573956 7720 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 03:44:39.578181 8268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:39.577746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:44939 (local address 127.0.0.1:57766)
0926 03:44:39.577927 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0926 03:44:39.578021 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.254:44939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260926 03:44:39.578189 7540 raft_consensus.cc:1261] T 00000000000000000000000000000000 P af7004004c624669ae81014cf255d63f [term 2 FOLLOWER]: Refusing update from remote peer 037743be8b9845979457b84234b6bdad: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260926 03:44:39.579267 8268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:39.578864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:44939 (local address 127.0.0.1:57774)
0926 03:44:39.579027 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0926 03:44:39.579108 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.254:44939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260926 03:44:39.579119 8271 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 037743be8b9845979457b84234b6bdad [LEADER]: Connected to new peer: Peer: permanent_uuid: "af7004004c624669ae81014cf255d63f" member_type: VOTER last_known_addr { host: "127.0.194.253" port: 37485 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260926 03:44:39.579433 7706 consensus_peers.cc:487] T 00000000000000000000000000000000 P 037743be8b9845979457b84234b6bdad -> Peer be6fefc6a90b41218affcf0df9459735 (127.0.194.254:44939): Couldn't send request to peer be6fefc6a90b41218affcf0df9459735. Status: Network error: Client connection negotiation failed: client connection to 127.0.194.254:44939: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260926 03:44:39.581934 8261 sys_catalog.cc:455] T 00000000000000000000000000000000 P af7004004c624669ae81014cf255d63f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 037743be8b9845979457b84234b6bdad. Latest consensus state: current_term: 2 leader_uuid: "037743be8b9845979457b84234b6bdad" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "be6fefc6a90b41218affcf0df9459735" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 44939 } } peers { permanent_uuid: "af7004004c624669ae81014cf255d63f" member_type: VOTER last_known_addr { host: "127.0.194.253" port: 37485 } attrs { promote: false } } peers { permanent_uuid: "037743be8b9845979457b84234b6bdad" member_type: VOTER last_known_addr { host: "127.0.194.252" port: 34225 } attrs { promote: false } } }
I20260926 03:44:39.582171 8261 sys_catalog.cc:458] T 00000000000000000000000000000000 P af7004004c624669ae81014cf255d63f [sys.catalog]: This master's current role is: FOLLOWER
I20260926 03:44:39.583650 8271 sys_catalog.cc:455] T 00000000000000000000000000000000 P 037743be8b9845979457b84234b6bdad [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "037743be8b9845979457b84234b6bdad" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "be6fefc6a90b41218affcf0df9459735" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 44939 } } peers { permanent_uuid: "af7004004c624669ae81014cf255d63f" member_type: VOTER last_known_addr { host: "127.0.194.253" port: 37485 } attrs { promote: false } } peers { permanent_uuid: "037743be8b9845979457b84234b6bdad" member_type: VOTER last_known_addr { host: "127.0.194.252" port: 34225 } attrs { promote: false } } }
I20260926 03:44:39.583927 8271 sys_catalog.cc:458] T 00000000000000000000000000000000 P 037743be8b9845979457b84234b6bdad [sys.catalog]: This master's current role is: LEADER
I20260926 03:44:39.584249 8261 sys_catalog.cc:455] T 00000000000000000000000000000000 P af7004004c624669ae81014cf255d63f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "037743be8b9845979457b84234b6bdad" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "be6fefc6a90b41218affcf0df9459735" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 44939 } } peers { permanent_uuid: "af7004004c624669ae81014cf255d63f" member_type: VOTER last_known_addr { host: "127.0.194.253" port: 37485 } attrs { promote: false } } peers { permanent_uuid: "037743be8b9845979457b84234b6bdad" member_type: VOTER last_known_addr { host: "127.0.194.252" port: 34225 } attrs { promote: false } } }
I20260926 03:44:39.584560 8261 sys_catalog.cc:458] T 00000000000000000000000000000000 P af7004004c624669ae81014cf255d63f [sys.catalog]: This master's current role is: FOLLOWER
I20260926 03:44:39.599159 8266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:39.592693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.195:34367 (local address 127.0.0.1:41432)
0926 03:44:39.593293 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0926 03:44:39.593318 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0926 03:44:39.593431 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:44:39.593821 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:44:39.593826 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:44:39.593854 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:44:39.594048 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:39.594059 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:39.595593 (+ 1534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:39.595599 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:39.596953 (+ 1354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:39.596998 (+ 45us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:39.598457 (+ 1459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:39.598463 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:39.598667 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:39.598692 (+ 25us) client_negotiation.cc:769] Sending connection context
0926 03:44:39.598834 (+ 142us) client_negotiation.cc:240] Negotiation successful
0926 03:44:39.598944 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":494,"mutex_wait_us":79}
I20260926 03:44:39.599836 8281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:39.592925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41432 (local address 127.0.194.195:34367)
0926 03:44:39.593471 (+ 546us) server_negotiation.cc:206] Beginning negotiation
0926 03:44:39.593512 (+ 41us) server_negotiation.cc:399] Waiting for connection header
0926 03:44:39.593531 (+ 19us) server_negotiation.cc:407] Connection header received
0926 03:44:39.593596 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:44:39.593601 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:44:39.593666 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:44:39.594171 (+ 505us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:44:39.594354 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:39.595429 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:39.597189 (+ 1760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:39.598295 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:39.598441 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:39.598545 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0926 03:44:39.599500 (+ 955us) server_negotiation.cc:299] Negotiation successful
0926 03:44:39.599642 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":421,"thread_start_us":258,"threads_started":1}
I20260926 03:44:39.600651 8123 tablet_service.cc:1467] Processing CreateTablet for tablet e36476e12d3f41a4803cda5604e65f7a (DEFAULT_TABLE table=test-table [id=8748956a318a4481a9b91673e8ee6308]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:44:39.601135 8123 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e36476e12d3f41a4803cda5604e65f7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:44:39.602079 8268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:39.592313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.194:42589 (local address 127.0.0.1:38682)
0926 03:44:39.592478 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0926 03:44:39.592499 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 03:44:39.592620 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:44:39.593596 (+ 976us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:44:39.593601 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:44:39.593628 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:44:39.593990 (+ 362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:39.594003 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:39.595576 (+ 1573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:39.595585 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:39.598402 (+ 2817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:39.598421 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:39.601067 (+ 2646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:39.601076 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:39.601352 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:39.601408 (+ 56us) client_negotiation.cc:769] Sending connection context
0926 03:44:39.601498 (+ 90us) client_negotiation.cc:240] Negotiation successful
0926 03:44:39.601685 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260926 03:44:39.603161 8280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:39.592532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38682 (local address 127.0.194.194:42589)
0926 03:44:39.593108 (+ 576us) server_negotiation.cc:206] Beginning negotiation
0926 03:44:39.593116 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:44:39.593139 (+ 23us) server_negotiation.cc:407] Connection header received
0926 03:44:39.593219 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:44:39.593226 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:44:39.593309 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:44:39.594135 (+ 826us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:44:39.594293 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:39.595419 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:39.598672 (+ 3253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:39.600781 (+ 2109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:39.602489 (+ 1708us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:39.602652 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0926 03:44:39.602715 (+ 63us) server_negotiation.cc:299] Negotiation successful
0926 03:44:39.602917 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":422,"thread_start_us":322,"threads_started":1}
I20260926 03:44:39.603883 7982 tablet_service.cc:1467] Processing CreateTablet for tablet e36476e12d3f41a4803cda5604e65f7a (DEFAULT_TABLE table=test-table [id=8748956a318a4481a9b91673e8ee6308]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:44:39.604372 7982 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e36476e12d3f41a4803cda5604e65f7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:44:39.607820 8284 tablet_bootstrap.cc:492] T e36476e12d3f41a4803cda5604e65f7a P 162b5f9a168f49db9ec3b1a5b0f0a526: Bootstrap starting.
I20260926 03:44:39.609579 8284 tablet_bootstrap.cc:654] T e36476e12d3f41a4803cda5604e65f7a P 162b5f9a168f49db9ec3b1a5b0f0a526: Neither blocks nor log segments found. Creating new log.
I20260926 03:44:39.609676 8282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:39.593008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.193:32927 (local address 127.0.0.1:52174)
0926 03:44:39.593671 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0926 03:44:39.593685 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 03:44:39.593806 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:44:39.603528 (+ 9722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:44:39.603535 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:44:39.603563 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:44:39.603893 (+ 330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:39.603907 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:39.605387 (+ 1480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:39.605395 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:39.607550 (+ 2155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:39.607563 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:39.608869 (+ 1306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:39.608874 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:39.609034 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:39.609058 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 03:44:39.609166 (+ 108us) client_negotiation.cc:240] Negotiation successful
0926 03:44:39.609303 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":545,"spinlock_wait_cycles":11520,"thread_start_us":837,"threads_started":1}
I20260926 03:44:39.609648 8283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:39.593766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52174 (local address 127.0.194.193:32927)
0926 03:44:39.603091 (+ 9325us) server_negotiation.cc:206] Beginning negotiation
0926 03:44:39.603099 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:44:39.603118 (+ 19us) server_negotiation.cc:407] Connection header received
0926 03:44:39.603220 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:44:39.603226 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:44:39.603384 (+ 158us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:44:39.603490 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:44:39.604063 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:39.605214 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:39.607769 (+ 2555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:39.608654 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:39.608805 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:39.608911 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0926 03:44:39.609194 (+ 283us) server_negotiation.cc:299] Negotiation successful
0926 03:44:39.609332 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9208,"thread_start_us":9117,"threads_started":1}
I20260926 03:44:39.610570 7835 tablet_service.cc:1467] Processing CreateTablet for tablet e36476e12d3f41a4803cda5604e65f7a (DEFAULT_TABLE table=test-table [id=8748956a318a4481a9b91673e8ee6308]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:44:39.611080 7835 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e36476e12d3f41a4803cda5604e65f7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:44:39.613137 8284 tablet_bootstrap.cc:492] T e36476e12d3f41a4803cda5604e65f7a P 162b5f9a168f49db9ec3b1a5b0f0a526: No bootstrap required, opened a new log
I20260926 03:44:39.613286 8284 ts_tablet_manager.cc:1397] T e36476e12d3f41a4803cda5604e65f7a P 162b5f9a168f49db9ec3b1a5b0f0a526: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260926 03:44:39.613924 8284 raft_consensus.cc:348] T e36476e12d3f41a4803cda5604e65f7a P 162b5f9a168f49db9ec3b1a5b0f0a526 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "162b5f9a168f49db9ec3b1a5b0f0a526" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 34367 } } peers { permanent_uuid: "949d66f722dd4b5c86409272002b87b3" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 42589 } } peers { permanent_uuid: "b08e95643f1a4377b8c2d42ac8762a70" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 32927 } }
I20260926 03:44:39.614104 8284 raft_consensus.cc:374] T e36476e12d3f41a4803cda5604e65f7a P 162b5f9a168f49db9ec3b1a5b0f0a526 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:44:39.614209 8284 raft_consensus.cc:729] T e36476e12d3f41a4803cda5604e65f7a P 162b5f9a168f49db9ec3b1a5b0f0a526 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 162b5f9a168f49db9ec3b1a5b0f0a526, State: Initialized, Role: FOLLOWER
I20260926 03:44:39.614470 8284 consensus_queue.cc:260] T e36476e12d3f41a4803cda5604e65f7a P 162b5f9a168f49db9ec3b1a5b0f0a526 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "162b5f9a168f49db9ec3b1a5b0f0a526" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 34367 } } peers { permanent_uuid: "949d66f722dd4b5c86409272002b87b3" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 42589 } } peers { permanent_uuid: "b08e95643f1a4377b8c2d42ac8762a70" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 32927 } }
I20260926 03:44:39.614862 8285 tablet_bootstrap.cc:492] T e36476e12d3f41a4803cda5604e65f7a P 949d66f722dd4b5c86409272002b87b3: Bootstrap starting.
I20260926 03:44:39.615947 8284 ts_tablet_manager.cc:1428] T e36476e12d3f41a4803cda5604e65f7a P 162b5f9a168f49db9ec3b1a5b0f0a526: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260926 03:44:39.618686 8188 heartbeater.cc:503] Master 127.0.194.252:34225 was elected leader, sending a full tablet report...
I20260926 03:44:39.619315 8289 tablet_bootstrap.cc:492] T e36476e12d3f41a4803cda5604e65f7a P b08e95643f1a4377b8c2d42ac8762a70: Bootstrap starting.
I20260926 03:44:39.620169 8285 tablet_bootstrap.cc:654] T e36476e12d3f41a4803cda5604e65f7a P 949d66f722dd4b5c86409272002b87b3: Neither blocks nor log segments found. Creating new log.
I20260926 03:44:39.621659 8289 tablet_bootstrap.cc:654] T e36476e12d3f41a4803cda5604e65f7a P b08e95643f1a4377b8c2d42ac8762a70: Neither blocks nor log segments found. Creating new log.
I20260926 03:44:39.621912 8288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:39.615532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:44939 (local address 127.0.194.195:50451)
0926 03:44:39.621548 (+ 6016us) negotiation.cc:107] Waiting for socket to connect
0926 03:44:39.621667 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.254:44939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":5904,"thread_start_us":5788,"threads_started":1}
I20260926 03:44:39.623063 8288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:39.622548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:44939 (local address 127.0.194.195:41883)
0926 03:44:39.622887 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0926 03:44:39.622959 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.254:44939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":236}
I20260926 03:44:39.623461 8285 tablet_bootstrap.cc:492] T e36476e12d3f41a4803cda5604e65f7a P 949d66f722dd4b5c86409272002b87b3: No bootstrap required, opened a new log
I20260926 03:44:39.623622 8285 ts_tablet_manager.cc:1397] T e36476e12d3f41a4803cda5604e65f7a P 949d66f722dd4b5c86409272002b87b3: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.000s
I20260926 03:44:39.623888 8289 tablet_bootstrap.cc:492] T e36476e12d3f41a4803cda5604e65f7a P b08e95643f1a4377b8c2d42ac8762a70: No bootstrap required, opened a new log
I20260926 03:44:39.624033 8289 ts_tablet_manager.cc:1397] T e36476e12d3f41a4803cda5604e65f7a P b08e95643f1a4377b8c2d42ac8762a70: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260926 03:44:39.624258 8285 raft_consensus.cc:348] T e36476e12d3f41a4803cda5604e65f7a P 949d66f722dd4b5c86409272002b87b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "162b5f9a168f49db9ec3b1a5b0f0a526" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 34367 } } peers { permanent_uuid: "949d66f722dd4b5c86409272002b87b3" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 42589 } } peers { permanent_uuid: "b08e95643f1a4377b8c2d42ac8762a70" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 32927 } }
I20260926 03:44:39.624447 8285 raft_consensus.cc:374] T e36476e12d3f41a4803cda5604e65f7a P 949d66f722dd4b5c86409272002b87b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:44:39.624517 8285 raft_consensus.cc:729] T e36476e12d3f41a4803cda5604e65f7a P 949d66f722dd4b5c86409272002b87b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 949d66f722dd4b5c86409272002b87b3, State: Initialized, Role: FOLLOWER
I20260926 03:44:39.624716 8285 consensus_queue.cc:260] T e36476e12d3f41a4803cda5604e65f7a P 949d66f722dd4b5c86409272002b87b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "162b5f9a168f49db9ec3b1a5b0f0a526" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 34367 } } peers { permanent_uuid: "949d66f722dd4b5c86409272002b87b3" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 42589 } } peers { permanent_uuid: "b08e95643f1a4377b8c2d42ac8762a70" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 32927 } }
I20260926 03:44:39.624881 8289 raft_consensus.cc:348] T e36476e12d3f41a4803cda5604e65f7a P b08e95643f1a4377b8c2d42ac8762a70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "162b5f9a168f49db9ec3b1a5b0f0a526" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 34367 } } peers { permanent_uuid: "949d66f722dd4b5c86409272002b87b3" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 42589 } } peers { permanent_uuid: "b08e95643f1a4377b8c2d42ac8762a70" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 32927 } }
I20260926 03:44:39.625126 8289 raft_consensus.cc:374] T e36476e12d3f41a4803cda5604e65f7a P b08e95643f1a4377b8c2d42ac8762a70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:44:39.625238 8289 raft_consensus.cc:729] T e36476e12d3f41a4803cda5604e65f7a P b08e95643f1a4377b8c2d42ac8762a70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b08e95643f1a4377b8c2d42ac8762a70, State: Initialized, Role: FOLLOWER
I20260926 03:44:39.625491 8289 consensus_queue.cc:260] T e36476e12d3f41a4803cda5604e65f7a P b08e95643f1a4377b8c2d42ac8762a70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "162b5f9a168f49db9ec3b1a5b0f0a526" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 34367 } } peers { permanent_uuid: "949d66f722dd4b5c86409272002b87b3" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 42589 } } peers { permanent_uuid: "b08e95643f1a4377b8c2d42ac8762a70" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 32927 } }
I20260926 03:44:39.625622 8285 ts_tablet_manager.cc:1428] T e36476e12d3f41a4803cda5604e65f7a P 949d66f722dd4b5c86409272002b87b3: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260926 03:44:39.627796 8047 heartbeater.cc:503] Master 127.0.194.252:34225 was elected leader, sending a full tablet report...
I20260926 03:44:39.631649 8294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:39.626261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:44939 (local address 127.0.194.194:35213)
0926 03:44:39.631326 (+ 5065us) negotiation.cc:107] Waiting for socket to connect
0926 03:44:39.631451 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.254:44939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":4913,"thread_start_us":4777,"threads_started":1}
I20260926 03:44:39.633174 8295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:39.632177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:44939 (local address 127.0.194.194:52989)
0926 03:44:39.632780 (+ 603us) negotiation.cc:107] Waiting for socket to connect
0926 03:44:39.632944 (+ 164us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.254:44939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":347,"threads_started":1}
I20260926 03:44:39.633225 8289 ts_tablet_manager.cc:1428] T e36476e12d3f41a4803cda5604e65f7a P b08e95643f1a4377b8c2d42ac8762a70: Time spent starting tablet: real 0.009s user 0.000s sys 0.009s
I20260926 03:44:39.634567 8296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:39.633453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:44939 (local address 127.0.194.193:58801)
0926 03:44:39.634157 (+ 704us) negotiation.cc:107] Waiting for socket to connect
0926 03:44:39.634302 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.254:44939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":573,"thread_start_us":269,"threads_started":1}
I20260926 03:44:39.635455 7900 heartbeater.cc:503] Master 127.0.194.252:34225 was elected leader, sending a full tablet report...
I20260926 03:44:39.636164 8296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:39.635812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:44939 (local address 127.0.194.193:35795)
0926 03:44:39.635957 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0926 03:44:39.636038 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.254:44939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260926 03:44:39.709575 8292 raft_consensus.cc:482] T e36476e12d3f41a4803cda5604e65f7a P 949d66f722dd4b5c86409272002b87b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 03:44:39.709808 8292 raft_consensus.cc:504] T e36476e12d3f41a4803cda5604e65f7a P 949d66f722dd4b5c86409272002b87b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "162b5f9a168f49db9ec3b1a5b0f0a526" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 34367 } } peers { permanent_uuid: "949d66f722dd4b5c86409272002b87b3" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 42589 } } peers { permanent_uuid: "b08e95643f1a4377b8c2d42ac8762a70" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 32927 } }
I20260926 03:44:39.710563 8292 leader_election.cc:288] T e36476e12d3f41a4803cda5604e65f7a P 949d66f722dd4b5c86409272002b87b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 162b5f9a168f49db9ec3b1a5b0f0a526 (127.0.194.195:34367), b08e95643f1a4377b8c2d42ac8762a70 (127.0.194.193:32927)
I20260926 03:44:39.711277 8143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e36476e12d3f41a4803cda5604e65f7a" candidate_uuid: "949d66f722dd4b5c86409272002b87b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "162b5f9a168f49db9ec3b1a5b0f0a526" is_pre_election: true
I20260926 03:44:39.711254 7855 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e36476e12d3f41a4803cda5604e65f7a" candidate_uuid: "949d66f722dd4b5c86409272002b87b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b08e95643f1a4377b8c2d42ac8762a70" is_pre_election: true
I20260926 03:44:39.711541 8143 raft_consensus.cc:2454] T e36476e12d3f41a4803cda5604e65f7a P 162b5f9a168f49db9ec3b1a5b0f0a526 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 949d66f722dd4b5c86409272002b87b3 in term 0.
I20260926 03:44:39.711558 7855 raft_consensus.cc:2454] T e36476e12d3f41a4803cda5604e65f7a P b08e95643f1a4377b8c2d42ac8762a70 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 949d66f722dd4b5c86409272002b87b3 in term 0.
I20260926 03:44:39.712051 7935 leader_election.cc:302] T e36476e12d3f41a4803cda5604e65f7a P 949d66f722dd4b5c86409272002b87b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 949d66f722dd4b5c86409272002b87b3, b08e95643f1a4377b8c2d42ac8762a70; no voters:
I20260926 03:44:39.712338 8292 raft_consensus.cc:2781] T e36476e12d3f41a4803cda5604e65f7a P 949d66f722dd4b5c86409272002b87b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 03:44:39.712463 8292 raft_consensus.cc:482] T e36476e12d3f41a4803cda5604e65f7a P 949d66f722dd4b5c86409272002b87b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 03:44:39.712559 8292 raft_consensus.cc:3037] T e36476e12d3f41a4803cda5604e65f7a P 949d66f722dd4b5c86409272002b87b3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:44:39.714574 8292 raft_consensus.cc:504] T e36476e12d3f41a4803cda5604e65f7a P 949d66f722dd4b5c86409272002b87b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "162b5f9a168f49db9ec3b1a5b0f0a526" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 34367 } } peers { permanent_uuid: "949d66f722dd4b5c86409272002b87b3" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 42589 } } peers { permanent_uuid: "b08e95643f1a4377b8c2d42ac8762a70" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 32927 } }
I20260926 03:44:39.715205 8292 leader_election.cc:288] T e36476e12d3f41a4803cda5604e65f7a P 949d66f722dd4b5c86409272002b87b3 [CANDIDATE]: Term 1 election: Requested vote from peers 162b5f9a168f49db9ec3b1a5b0f0a526 (127.0.194.195:34367), b08e95643f1a4377b8c2d42ac8762a70 (127.0.194.193:32927)
I20260926 03:44:39.715864 8143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e36476e12d3f41a4803cda5604e65f7a" candidate_uuid: "949d66f722dd4b5c86409272002b87b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "162b5f9a168f49db9ec3b1a5b0f0a526"
I20260926 03:44:39.715883 7855 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e36476e12d3f41a4803cda5604e65f7a" candidate_uuid: "949d66f722dd4b5c86409272002b87b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b08e95643f1a4377b8c2d42ac8762a70"
I20260926 03:44:39.716085 8143 raft_consensus.cc:3037] T e36476e12d3f41a4803cda5604e65f7a P 162b5f9a168f49db9ec3b1a5b0f0a526 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:44:39.716130 7855 raft_consensus.cc:3037] T e36476e12d3f41a4803cda5604e65f7a P b08e95643f1a4377b8c2d42ac8762a70 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:44:39.719086 7855 raft_consensus.cc:2454] T e36476e12d3f41a4803cda5604e65f7a P b08e95643f1a4377b8c2d42ac8762a70 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 949d66f722dd4b5c86409272002b87b3 in term 1.
I20260926 03:44:39.719131 8143 raft_consensus.cc:2454] T e36476e12d3f41a4803cda5604e65f7a P 162b5f9a168f49db9ec3b1a5b0f0a526 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 949d66f722dd4b5c86409272002b87b3 in term 1.
I20260926 03:44:39.719650 7935 leader_election.cc:302] T e36476e12d3f41a4803cda5604e65f7a P 949d66f722dd4b5c86409272002b87b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 162b5f9a168f49db9ec3b1a5b0f0a526, 949d66f722dd4b5c86409272002b87b3; no voters:
I20260926 03:44:39.720048 8292 raft_consensus.cc:2781] T e36476e12d3f41a4803cda5604e65f7a P 949d66f722dd4b5c86409272002b87b3 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:44:39.720743 8292 raft_consensus.cc:686] T e36476e12d3f41a4803cda5604e65f7a P 949d66f722dd4b5c86409272002b87b3 [term 1 LEADER]: Becoming Leader. State: Replica: 949d66f722dd4b5c86409272002b87b3, State: Running, Role: LEADER
I20260926 03:44:39.721088 8292 consensus_queue.cc:237] T e36476e12d3f41a4803cda5604e65f7a P 949d66f722dd4b5c86409272002b87b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "162b5f9a168f49db9ec3b1a5b0f0a526" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 34367 } } peers { permanent_uuid: "949d66f722dd4b5c86409272002b87b3" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 42589 } } peers { permanent_uuid: "b08e95643f1a4377b8c2d42ac8762a70" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 32927 } }
I20260926 03:44:39.721814 8295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:39.721324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:44939 (local address 127.0.194.194:33487)
0926 03:44:39.721565 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0926 03:44:39.721661 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.254:44939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134}
I20260926 03:44:39.724169 7719 catalog_manager.cc:5495] T e36476e12d3f41a4803cda5604e65f7a P 949d66f722dd4b5c86409272002b87b3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 949d66f722dd4b5c86409272002b87b3 (127.0.194.194). New cstate: current_term: 1 leader_uuid: "949d66f722dd4b5c86409272002b87b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "162b5f9a168f49db9ec3b1a5b0f0a526" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 34367 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "949d66f722dd4b5c86409272002b87b3" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 42589 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b08e95643f1a4377b8c2d42ac8762a70" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 32927 } health_report { overall_health: UNKNOWN } } }
I20260926 03:44:39.760682 8280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:39.756554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38686 (local address 127.0.194.194:42589)
0926 03:44:39.756738 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0926 03:44:39.756742 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:44:39.756758 (+ 16us) server_negotiation.cc:407] Connection header received
0926 03:44:39.756872 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:44:39.756875 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:44:39.756935 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:44:39.757016 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:44:39.757305 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:39.757999 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:39.759291 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:39.759680 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:39.759769 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:39.760120 (+ 351us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:44:39.760212 (+ 92us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:44:39.760276 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 03:44:39.760440 (+ 164us) server_negotiation.cc:299] Negotiation successful
0926 03:44:39.760496 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 03:44:39.763652 7855 raft_consensus.cc:1261] T e36476e12d3f41a4803cda5604e65f7a P b08e95643f1a4377b8c2d42ac8762a70 [term 1 FOLLOWER]: Refusing update from remote peer 949d66f722dd4b5c86409272002b87b3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:44:39.763772 8143 raft_consensus.cc:1261] T e36476e12d3f41a4803cda5604e65f7a P 162b5f9a168f49db9ec3b1a5b0f0a526 [term 1 FOLLOWER]: Refusing update from remote peer 949d66f722dd4b5c86409272002b87b3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:44:39.764338 8297 consensus_queue.cc:1035] T e36476e12d3f41a4803cda5604e65f7a P 949d66f722dd4b5c86409272002b87b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b08e95643f1a4377b8c2d42ac8762a70" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:44:39.764703 8292 consensus_queue.cc:1035] T e36476e12d3f41a4803cda5604e65f7a P 949d66f722dd4b5c86409272002b87b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "162b5f9a168f49db9ec3b1a5b0f0a526" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 34367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:44:39.767153 8288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:39.766499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:44939 (local address 127.0.194.195:52657)
0926 03:44:39.766773 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0926 03:44:39.766917 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.254:44939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115}
I20260926 03:44:39.767655 8296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:39.766955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:44939 (local address 127.0.194.193:52015)
0926 03:44:39.767241 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0926 03:44:39.767391 (+ 150us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.254:44939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148}
I20260926 03:44:39.772964 8302 mvcc.cc:205] Tried to move back new op lower bound from 7333454969905958912 to 7333454969740075008. Current Snapshot: MvccSnapshot[applied={T|T < 7333454969905958912 or (T in {7333454969905958912})}]
I20260926 03:44:39.774150 8295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:39.772738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:44939 (local address 127.0.194.194:32935)
0926 03:44:39.773796 (+ 1058us) negotiation.cc:107] Waiting for socket to connect
0926 03:44:39.773925 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.254:44939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":896}
I20260926 03:44:39.774791 8296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:39.773492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:44939 (local address 127.0.194.193:50033)
0926 03:44:39.774481 (+ 989us) negotiation.cc:107] Waiting for socket to connect
0926 03:44:39.774590 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.254:44939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":834}
I20260926 03:44:39.780925 8295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:39.779876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:44939 (local address 127.0.194.194:60003)
0926 03:44:39.780608 (+ 732us) negotiation.cc:107] Waiting for socket to connect
0926 03:44:39.780720 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.254:44939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":603}
I20260926 03:44:39.783329 8288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:39.782814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:44939 (local address 127.0.194.195:48045)
0926 03:44:39.783077 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0926 03:44:39.783161 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.254:44939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140}
I20260926 03:44:39.786170 7720 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53046:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:44:39.786401 7720 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53046:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:44:39.792428 7720 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 037743be8b9845979457b84234b6bdad: Sending DeleteTablet for 3 replicas of tablet e36476e12d3f41a4803cda5604e65f7a
I20260926 03:44:39.793280 8123 tablet_service.cc:1514] Processing DeleteTablet for tablet e36476e12d3f41a4803cda5604e65f7a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:44:39 UTC) from {username='oryx', principal='oryx/127.0.194.252@KRBTEST.COM'} at 127.0.0.1:41432
I20260926 03:44:39.793416 7982 tablet_service.cc:1514] Processing DeleteTablet for tablet e36476e12d3f41a4803cda5604e65f7a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:44:39 UTC) from {username='oryx', principal='oryx/127.0.194.252@KRBTEST.COM'} at 127.0.0.1:38682
I20260926 03:44:39.793501 7835 tablet_service.cc:1514] Processing DeleteTablet for tablet e36476e12d3f41a4803cda5604e65f7a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:44:39 UTC) from {username='oryx', principal='oryx/127.0.194.252@KRBTEST.COM'} at 127.0.0.1:52174
I20260926 03:44:39.793955 8309 tablet_replica.cc:330] T e36476e12d3f41a4803cda5604e65f7a P 162b5f9a168f49db9ec3b1a5b0f0a526: stopping tablet replica
I20260926 03:44:39.794152 8309 raft_consensus.cc:2229] T e36476e12d3f41a4803cda5604e65f7a P 162b5f9a168f49db9ec3b1a5b0f0a526 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:44:39.794373 8311 tablet_replica.cc:330] T e36476e12d3f41a4803cda5604e65f7a P b08e95643f1a4377b8c2d42ac8762a70: stopping tablet replica
I20260926 03:44:39.794448 8309 raft_consensus.cc:2258] T e36476e12d3f41a4803cda5604e65f7a P 162b5f9a168f49db9ec3b1a5b0f0a526 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:44:39.794723 8311 raft_consensus.cc:2229] T e36476e12d3f41a4803cda5604e65f7a P b08e95643f1a4377b8c2d42ac8762a70 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:44:39.795137 779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu pid 7707
I20260926 03:44:39.795186 8311 raft_consensus.cc:2258] T e36476e12d3f41a4803cda5604e65f7a P b08e95643f1a4377b8c2d42ac8762a70 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:44:39.796716 8311 ts_tablet_manager.cc:1905] T e36476e12d3f41a4803cda5604e65f7a P b08e95643f1a4377b8c2d42ac8762a70: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:44:39.797459 8309 ts_tablet_manager.cc:1905] T e36476e12d3f41a4803cda5604e65f7a P 162b5f9a168f49db9ec3b1a5b0f0a526: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:44:39.803413 8309 ts_tablet_manager.cc:1918] T e36476e12d3f41a4803cda5604e65f7a P 162b5f9a168f49db9ec3b1a5b0f0a526: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:44:39.803556 8309 log.cc:1196] T e36476e12d3f41a4803cda5604e65f7a P 162b5f9a168f49db9ec3b1a5b0f0a526: Deleting WAL directory at /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/minicluster-data/ts-2/wal/wals/e36476e12d3f41a4803cda5604e65f7a
I20260926 03:44:39.804013 8309 ts_tablet_manager.cc:1939] T e36476e12d3f41a4803cda5604e65f7a P 162b5f9a168f49db9ec3b1a5b0f0a526: Deleting consensus metadata
I20260926 03:44:39.805226 7703 catalog_manager.cc:4841] TS 162b5f9a168f49db9ec3b1a5b0f0a526 (127.0.194.195:34367): tablet e36476e12d3f41a4803cda5604e65f7a (table test-table [id=8748956a318a4481a9b91673e8ee6308]) successfully deleted
I20260926 03:44:39.805356 8311 ts_tablet_manager.cc:1918] T e36476e12d3f41a4803cda5604e65f7a P b08e95643f1a4377b8c2d42ac8762a70: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:44:39.805533 8311 log.cc:1196] T e36476e12d3f41a4803cda5604e65f7a P b08e95643f1a4377b8c2d42ac8762a70: Deleting WAL directory at /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/minicluster-data/ts-0/wal/wals/e36476e12d3f41a4803cda5604e65f7a
I20260926 03:44:39.806303 8311 ts_tablet_manager.cc:1939] T e36476e12d3f41a4803cda5604e65f7a P b08e95643f1a4377b8c2d42ac8762a70: Deleting consensus metadata
I20260926 03:44:39.807247 8310 tablet_replica.cc:330] T e36476e12d3f41a4803cda5604e65f7a P 949d66f722dd4b5c86409272002b87b3: stopping tablet replica
I20260926 03:44:39.807462 8310 raft_consensus.cc:2229] T e36476e12d3f41a4803cda5604e65f7a P 949d66f722dd4b5c86409272002b87b3 [term 1 LEADER]: Raft consensus shutting down.
I20260926 03:44:39.807577 7703 catalog_manager.cc:4841] TS b08e95643f1a4377b8c2d42ac8762a70 (127.0.194.193:32927): tablet e36476e12d3f41a4803cda5604e65f7a (table test-table [id=8748956a318a4481a9b91673e8ee6308]) successfully deleted
I20260926 03:44:39.807912 8310 raft_consensus.cc:2258] T e36476e12d3f41a4803cda5604e65f7a P 949d66f722dd4b5c86409272002b87b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:44:39.808959 8310 ts_tablet_manager.cc:1905] T e36476e12d3f41a4803cda5604e65f7a P 949d66f722dd4b5c86409272002b87b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:44:39.814970 8310 ts_tablet_manager.cc:1918] T e36476e12d3f41a4803cda5604e65f7a P 949d66f722dd4b5c86409272002b87b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:44:39.815122 8310 log.cc:1196] T e36476e12d3f41a4803cda5604e65f7a P 949d66f722dd4b5c86409272002b87b3: Deleting WAL directory at /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790394221564982-779-0/minicluster-data/ts-1/wal/wals/e36476e12d3f41a4803cda5604e65f7a
I20260926 03:44:39.815603 8310 ts_tablet_manager.cc:1939] T e36476e12d3f41a4803cda5604e65f7a P 949d66f722dd4b5c86409272002b87b3: Deleting consensus metadata
I20260926 03:44:39.816208 8283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:39.795877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52188 (local address 127.0.194.193:32927)
0926 03:44:39.796076 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0926 03:44:39.796084 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:44:39.799223 (+ 3139us) server_negotiation.cc:407] Connection header received
0926 03:44:39.799470 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:44:39.799476 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:44:39.799572 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:44:39.799638 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:44:39.801548 (+ 1910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:39.802682 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:39.805186 (+ 2504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:39.805798 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:39.805873 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:39.809618 (+ 3745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:44:39.809634 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:44:39.809650 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:44:39.809685 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:44:39.813314 (+ 3629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:39.813985 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:39.813996 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:39.814000 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:39.814157 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:39.814522 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:39.814527 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:39.814530 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:39.815028 (+ 498us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:44:39.815348 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0926 03:44:39.815656 (+ 308us) server_negotiation.cc:299] Negotiation successful
0926 03:44:39.815897 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 03:44:39.816752 7704 catalog_manager.cc:4841] TS 949d66f722dd4b5c86409272002b87b3 (127.0.194.194:42589): tablet e36476e12d3f41a4803cda5604e65f7a (table test-table [id=8748956a318a4481a9b91673e8ee6308]) successfully deleted
I20260926 03:44:39.816943 7895 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:44:39.915997 779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu with pid 7707
I20260926 03:44:39.934989 779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu pid 7912
I20260926 03:44:39.948693 8280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:39.935728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38700 (local address 127.0.194.194:42589)
0926 03:44:39.935924 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0926 03:44:39.935928 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:44:39.935949 (+ 21us) server_negotiation.cc:407] Connection header received
0926 03:44:39.936122 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:44:39.936126 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:44:39.936198 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:44:39.936283 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:44:39.937798 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:39.938587 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:39.940183 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:39.940570 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:39.940672 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:39.943318 (+ 2646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:44:39.943333 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:44:39.943349 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:44:39.943385 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:44:39.946131 (+ 2746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:39.946790 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:39.946806 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:39.946808 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:39.946950 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:39.947252 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:39.947256 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:39.947257 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:39.947591 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:44:39.947870 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0926 03:44:39.948174 (+ 304us) server_negotiation.cc:299] Negotiation successful
0926 03:44:39.948338 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260926 03:44:39.949393 8042 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:44:40.071759 779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu with pid 7912
I20260926 03:44:40.093827 779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu pid 8060
I20260926 03:44:40.106573 8281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:40.094595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41448 (local address 127.0.194.195:34367)
0926 03:44:40.094838 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0926 03:44:40.094842 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:44:40.094860 (+ 18us) server_negotiation.cc:407] Connection header received
0926 03:44:40.095030 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:44:40.095034 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:44:40.095104 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:44:40.095188 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:44:40.096478 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:40.097147 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:40.098401 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:40.098869 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:40.098971 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:40.101436 (+ 2465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:44:40.101449 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:44:40.101465 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:44:40.101498 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:44:40.104260 (+ 2762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:40.104890 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:40.104897 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:40.104899 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:40.105030 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:40.105328 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:40.105331 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:40.105333 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:40.105669 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:44:40.105921 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0926 03:44:40.106168 (+ 247us) server_negotiation.cc:299] Negotiation successful
0926 03:44:40.106332 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 03:44:40.107422 8183 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:44:40.170706 8316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:40.170025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:44939 (local address 127.0.0.1:57778)
0926 03:44:40.170443 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0926 03:44:40.170535 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.254:44939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":225,"threads_started":1}
I20260926 03:44:40.173955 8316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:40.173460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:44939 (local address 127.0.0.1:57790)
0926 03:44:40.173739 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0926 03:44:40.173826 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.254:44939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156}
I20260926 03:44:40.202394 779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu with pid 8060
I20260926 03:44:40.220199 779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu pid 7394
I20260926 03:44:40.221311 7561 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:44:40.336827 779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu with pid 7394
I20260926 03:44:40.358069 779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu pid 7494
I20260926 03:44:40.359261 7750 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:44:40.477228 8316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:40.359515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.253:37485 (local address 127.0.0.1:43966)
0926 03:44:40.359708 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0926 03:44:40.359783 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.253:37485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260926 03:44:40.477748 779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu with pid 7494
2026-09-26T03:44:40Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (9452 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790394221564982-779-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:44:40 dist-test-slave-n2th krb5kdc[8322](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:44:40 dist-test-slave-n2th krb5kdc[8322](info): set up 2 sockets
Sep 26 03:44:40 dist-test-slave-n2th krb5kdc[8322](info): commencing operation
krb5kdc: starting...
W20260926 03:44:41.579851 779 mini_kdc.cc:121] Time spent starting KDC: real 1.056s user 0.003s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790394221564982-779-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790394221564982-779-0/krb5kdc/test-user.keytab.
Sep 26 03:44:41 dist-test-slave-n2th krb5kdc[8322](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394281, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:44:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:44:41Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.194.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.194.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.194.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790394221564982-779-0/krb5kdc/oryx_127.0.194.254.keytab.
Entry for principal oryx/127.0.194.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790394221564982-779-0/krb5kdc/oryx_127.0.194.254.keytab.
I20260926 03:44:41.797447 779 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
/tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790394221564982-779-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790394221564982-779-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790394221564982-779-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790394221564982-779-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.194.254:34071
--webserver_interface=127.0.194.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790394221564982-779-0/krb5kdc/oryx_127.0.194.254.keytab
--principal=oryx/127.0.194.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.194.212:41615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.194.254:34071
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790394221564982-779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790394221564982-779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790394221564982-779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:44:42.115206 8335 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:44:42.115551 8335 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:44:42.115701 8335 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:44:42.122972 8335 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:44:42.123107 8335 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260926 03:44:42.123194 8335 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:44:42.123308 8335 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:44:42.123395 8335 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:44:42.123497 8335 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:44:42.133366 8335 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.194.212:41615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790394221564982-779-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790394221564982-779-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.194.254:34071
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790394221564982-779-0/krb5kdc/oryx_127.0.194.254.keytab
--principal=oryx/127.0.194.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.194.254:34071
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790394221564982-779-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.194.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790394221564982-779-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:44:42.134698 8335 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:44:42.136060 8335 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:44:42.147732 8340 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:44:42.147781 8341 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:44:42.147689 8343 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:44:42.154070 8335 server_base.cc:1034] running on GCE node
I20260926 03:44:42.154901 8335 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:44:42.156229 8335 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:44:42.157465 8335 hybrid_clock.cc:648] HybridClock initialized: now 1790394282157389 us; error 33 us; skew 500 ppm
I20260926 03:44:42.157750 8335 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:44:42 dist-test-slave-n2th krb5kdc[8322](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394282, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.194.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:44:42.160780 8335 init.cc:377] Logged in from keytab as oryx/127.0.194.254@KRBTEST.COM (short username oryx)
I20260926 03:44:42.162289 8335 webserver.cc:460] Webserver started at http://127.0.194.254:46623/ using document root <none> and password file <none>
I20260926 03:44:42.163061 8335 fs_manager.cc:359] Metadata directory not provided
I20260926 03:44:42.163151 8335 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:44:42.163420 8335 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:44:42.166299 8335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790394221564982-779-0/minicluster-data/master-0/data/instance:
uuid: "f8f4c2ad1ed14a06a9cac074539f400f"
format_stamp: "Formatted at 2026-09-26 03:44:42 on dist-test-slave-n2th"
server_key: "8e5aa0afb636cad3d6708c19ea1fae61"
server_key_iv: "9f5fda834a2db795073b18a3b0627041"
server_key_version: "encryptionkey@0"
I20260926 03:44:42.167122 8335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790394221564982-779-0/minicluster-data/master-0/wal/instance:
uuid: "f8f4c2ad1ed14a06a9cac074539f400f"
format_stamp: "Formatted at 2026-09-26 03:44:42 on dist-test-slave-n2th"
server_key: "8e5aa0afb636cad3d6708c19ea1fae61"
server_key_iv: "9f5fda834a2db795073b18a3b0627041"
server_key_version: "encryptionkey@0"
I20260926 03:44:42.172174 8335 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260926 03:44:42.175462 8350 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:44:42.176766 8335 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 03:44:42.176946 8335 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790394221564982-779-0/minicluster-data/master-0/data,/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790394221564982-779-0/minicluster-data/master-0/wal
uuid: "f8f4c2ad1ed14a06a9cac074539f400f"
format_stamp: "Formatted at 2026-09-26 03:44:42 on dist-test-slave-n2th"
server_key: "8e5aa0afb636cad3d6708c19ea1fae61"
server_key_iv: "9f5fda834a2db795073b18a3b0627041"
server_key_version: "encryptionkey@0"
I20260926 03:44:42.177160 8335 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790394221564982-779-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790394221564982-779-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790394221564982-779-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:44:42.194962 8335 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:44:42.199450 8335 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:44:42.199757 8335 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:44:42.220180 8335 rpc_server.cc:307] RPC server started. Bound to: 127.0.194.254:34071
I20260926 03:44:42.220211 8411 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.194.254:34071 every 8 connection(s)
I20260926 03:44:42.221585 8335 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790394221564982-779-0/minicluster-data/master-0/data/info.pb
I20260926 03:44:42.225533 8412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:44:42.227862 779 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu as pid 8335
I20260926 03:44:42.228055 779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790394221564982-779-0/minicluster-data/master-0/wal/instance
I20260926 03:44:42.228700 779 external_mini_cluster.cc:1428] Setting key a4708a859c1ce0f9fc5aa633c035844b
I20260926 03:44:42.238327 8412 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f8f4c2ad1ed14a06a9cac074539f400f: Bootstrap starting.
I20260926 03:44:42.241933 8412 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f8f4c2ad1ed14a06a9cac074539f400f: Neither blocks nor log segments found. Creating new log.
I20260926 03:44:42.243016 8412 log.cc:824] T 00000000000000000000000000000000 P f8f4c2ad1ed14a06a9cac074539f400f: Log is configured to *not* fsync() on all Append() calls
Sep 26 03:44:42 dist-test-slave-n2th krb5kdc[8322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394281, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.194.254@KRBTEST.COM
I20260926 03:44:42.246253 8412 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f8f4c2ad1ed14a06a9cac074539f400f: No bootstrap required, opened a new log
I20260926 03:44:42.253885 8415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:42.230720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56308 (local address 127.0.194.254:34071)
0926 03:44:42.231554 (+ 834us) server_negotiation.cc:206] Beginning negotiation
0926 03:44:42.231571 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 03:44:42.231619 (+ 48us) server_negotiation.cc:407] Connection header received
0926 03:44:42.234091 (+ 2472us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:44:42.234128 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:44:42.234698 (+ 570us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:44:42.235172 (+ 474us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:44:42.236407 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:42.238312 (+ 1905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:42.239702 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:42.240799 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:42.240983 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:42.244975 (+ 3992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:44:42.244988 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:44:42.245010 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:44:42.245052 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:44:42.249126 (+ 4074us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:42.249864 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:42.249873 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:42.249882 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:42.250098 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:42.250446 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:42.250450 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:42.250453 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:42.251202 (+ 749us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:44:42.251476 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0926 03:44:42.251893 (+ 417us) server_negotiation.cc:299] Negotiation successful
0926 03:44:42.252373 (+ 480us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":565,"thread_start_us":357,"threads_started":1}
I20260926 03:44:42.256932 8412 raft_consensus.cc:348] T 00000000000000000000000000000000 P f8f4c2ad1ed14a06a9cac074539f400f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8f4c2ad1ed14a06a9cac074539f400f" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 34071 } }
I20260926 03:44:42.257292 8412 raft_consensus.cc:374] T 00000000000000000000000000000000 P f8f4c2ad1ed14a06a9cac074539f400f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:44:42.257366 8412 raft_consensus.cc:729] T 00000000000000000000000000000000 P f8f4c2ad1ed14a06a9cac074539f400f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8f4c2ad1ed14a06a9cac074539f400f, State: Initialized, Role: FOLLOWER
I20260926 03:44:42.257908 8412 consensus_queue.cc:260] T 00000000000000000000000000000000 P f8f4c2ad1ed14a06a9cac074539f400f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8f4c2ad1ed14a06a9cac074539f400f" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 34071 } }
I20260926 03:44:42.258101 8412 raft_consensus.cc:388] T 00000000000000000000000000000000 P f8f4c2ad1ed14a06a9cac074539f400f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:44:42.258194 8412 raft_consensus.cc:482] T 00000000000000000000000000000000 P f8f4c2ad1ed14a06a9cac074539f400f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:44:42.258311 8412 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f8f4c2ad1ed14a06a9cac074539f400f [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:44:42.260461 8412 raft_consensus.cc:504] T 00000000000000000000000000000000 P f8f4c2ad1ed14a06a9cac074539f400f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8f4c2ad1ed14a06a9cac074539f400f" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 34071 } }
I20260926 03:44:42.260906 8412 leader_election.cc:302] T 00000000000000000000000000000000 P f8f4c2ad1ed14a06a9cac074539f400f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f8f4c2ad1ed14a06a9cac074539f400f; no voters:
I20260926 03:44:42.261459 8412 leader_election.cc:288] T 00000000000000000000000000000000 P f8f4c2ad1ed14a06a9cac074539f400f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:44:42.261627 8417 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f8f4c2ad1ed14a06a9cac074539f400f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:44:42.262351 8417 raft_consensus.cc:686] T 00000000000000000000000000000000 P f8f4c2ad1ed14a06a9cac074539f400f [term 1 LEADER]: Becoming Leader. State: Replica: f8f4c2ad1ed14a06a9cac074539f400f, State: Running, Role: LEADER
I20260926 03:44:42.262977 8417 consensus_queue.cc:237] T 00000000000000000000000000000000 P f8f4c2ad1ed14a06a9cac074539f400f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8f4c2ad1ed14a06a9cac074539f400f" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 34071 } }
I20260926 03:44:42.263784 8412 sys_catalog.cc:564] T 00000000000000000000000000000000 P f8f4c2ad1ed14a06a9cac074539f400f [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:44:42.265698 8419 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8f4c2ad1ed14a06a9cac074539f400f [sys.catalog]: SysCatalogTable state changed. Reason: New leader f8f4c2ad1ed14a06a9cac074539f400f. Latest consensus state: current_term: 1 leader_uuid: "f8f4c2ad1ed14a06a9cac074539f400f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8f4c2ad1ed14a06a9cac074539f400f" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 34071 } } }
I20260926 03:44:42.266155 8418 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8f4c2ad1ed14a06a9cac074539f400f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f8f4c2ad1ed14a06a9cac074539f400f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8f4c2ad1ed14a06a9cac074539f400f" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 34071 } } }
I20260926 03:44:42.266355 8419 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8f4c2ad1ed14a06a9cac074539f400f [sys.catalog]: This master's current role is: LEADER
I20260926 03:44:42.266443 8418 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8f4c2ad1ed14a06a9cac074539f400f [sys.catalog]: This master's current role is: LEADER
I20260926 03:44:42.268656 8425 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:44:42.274034 8425 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:44:42.282814 8425 catalog_manager.cc:1347] Generated new cluster ID: 75569805bd854d6cb6004139bac88bab
I20260926 03:44:42.282931 8425 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:44:42.308442 8425 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:44:42.309823 8425 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:44:42.323911 8425 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f8f4c2ad1ed14a06a9cac074539f400f: Generated new TSK 0
I20260926 03:44:42.325012 8425 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.194.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.194.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.194.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790394221564982-779-0/krb5kdc/oryx_127.0.194.193.keytab.
Entry for principal oryx/127.0.194.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790394221564982-779-0/krb5kdc/oryx_127.0.194.193.keytab.
I20260926 03:44:42.385532 779 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
/tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790394221564982-779-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790394221564982-779-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790394221564982-779-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790394221564982-779-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.194.193:0
--local_ip_for_outbound_sockets=127.0.194.193
--webserver_interface=127.0.194.193
--webserver_port=0
--tserver_master_addrs=127.0.194.254:34071
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790394221564982-779-0/krb5kdc/oryx_127.0.194.193.keytab
--principal=oryx/127.0.194.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.194.212:41615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790394221564982-779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790394221564982-779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790394221564982-779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:44:42.712574 8438 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:44:42.712895 8438 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:44:42.713063 8438 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:44:42.720307 8438 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:44:42.720458 8438 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:44:42.720616 8438 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 03:44:42.720752 8438 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.194.193
I20260926 03:44:42.730396 8438 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.194.212:41615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790394221564982-779-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790394221564982-779-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790394221564982-779-0/krb5kdc/oryx_127.0.194.193.keytab
--principal=oryx/127.0.194.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.194.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790394221564982-779-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.194.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.194.254:34071
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.194.193
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790394221564982-779-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:44:42.731743 8438 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:44:42.733042 8438 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:44:42.745589 8446 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:44:42.745707 8444 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:44:42.745676 8443 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:44:42.746665 8438 server_base.cc:1034] running on GCE node
I20260926 03:44:42.747241 8438 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:44:42.748085 8438 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:44:42.749289 8438 hybrid_clock.cc:648] HybridClock initialized: now 1790394282749232 us; error 31 us; skew 500 ppm
I20260926 03:44:42.749575 8438 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:44:42 dist-test-slave-n2th krb5kdc[8322](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394282, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.194.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:44:42.752727 8438 init.cc:377] Logged in from keytab as oryx/127.0.194.193@KRBTEST.COM (short username oryx)
I20260926 03:44:42.754086 8438 webserver.cc:460] Webserver started at http://127.0.194.193:35943/ using document root <none> and password file <none>
I20260926 03:44:42.754694 8438 fs_manager.cc:359] Metadata directory not provided
I20260926 03:44:42.754841 8438 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:44:42.755187 8438 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:44:42.757941 8438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790394221564982-779-0/minicluster-data/ts-0/data/instance:
uuid: "ed1e5938558345edaccaa79fd9ed26b9"
format_stamp: "Formatted at 2026-09-26 03:44:42 on dist-test-slave-n2th"
server_key: "b54a68915f311ed9c7bab7e09b873286"
server_key_iv: "94c9ae3dbf816231f1c195a5318552e7"
server_key_version: "encryptionkey@0"
I20260926 03:44:42.758716 8438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790394221564982-779-0/minicluster-data/ts-0/wal/instance:
uuid: "ed1e5938558345edaccaa79fd9ed26b9"
format_stamp: "Formatted at 2026-09-26 03:44:42 on dist-test-slave-n2th"
server_key: "b54a68915f311ed9c7bab7e09b873286"
server_key_iv: "94c9ae3dbf816231f1c195a5318552e7"
server_key_version: "encryptionkey@0"
I20260926 03:44:42.763651 8438 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260926 03:44:42.767043 8453 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:44:42.768347 8438 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 03:44:42.768514 8438 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790394221564982-779-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790394221564982-779-0/minicluster-data/ts-0/wal
uuid: "ed1e5938558345edaccaa79fd9ed26b9"
format_stamp: "Formatted at 2026-09-26 03:44:42 on dist-test-slave-n2th"
server_key: "b54a68915f311ed9c7bab7e09b873286"
server_key_iv: "94c9ae3dbf816231f1c195a5318552e7"
server_key_version: "encryptionkey@0"
I20260926 03:44:42.768759 8438 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790394221564982-779-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790394221564982-779-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790394221564982-779-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:44:42.807227 8438 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:44:42.811651 8438 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:44:42.811883 8438 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:44:42.814450 8438 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:44:42.814538 8438 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:44:42.814718 8438 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:44:42.814833 8438 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
Sep 26 03:44:42 dist-test-slave-n2th krb5kdc[8322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394282, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.194.193@KRBTEST.COM for oryx/127.0.194.254@KRBTEST.COM
I20260926 03:44:42.842451 8462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:42.815769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.193:36155 (local address 127.0.194.254:34071)
0926 03:44:42.816254 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0926 03:44:42.816262 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:44:42.819522 (+ 3260us) server_negotiation.cc:407] Connection header received
0926 03:44:42.822864 (+ 3342us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:44:42.822870 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:44:42.822973 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:44:42.823106 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:44:42.826061 (+ 2955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:42.827134 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:42.829954 (+ 2820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:42.830563 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:42.830696 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:42.835118 (+ 4422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:44:42.835127 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:44:42.835135 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:44:42.835188 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:44:42.838288 (+ 3100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:42.839507 (+ 1219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:42.839515 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:42.839521 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:42.839678 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:42.840321 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:42.840333 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:42.840335 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:42.840668 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:44:42.840825 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0926 03:44:42.841853 (+ 1028us) server_negotiation.cc:299] Negotiation successful
0926 03:44:42.842086 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":234,"threads_started":1}
I20260926 03:44:42.844022 8463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:42.815783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:34071 (local address 127.0.194.193:36155)
0926 03:44:42.819347 (+ 3564us) negotiation.cc:107] Waiting for socket to connect
0926 03:44:42.819393 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 03:44:42.822504 (+ 3111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:44:42.823833 (+ 1329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:44:42.823851 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:44:42.824519 (+ 668us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:44:42.825533 (+ 1014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:42.825561 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:42.827322 (+ 1761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:42.827326 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:42.829795 (+ 2469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:42.829808 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:42.830836 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:42.830843 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:42.831111 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:42.832041 (+ 930us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:44:42.832067 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:44:42.834859 (+ 2792us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:44:42.838444 (+ 3585us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:44:42.838458 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:44:42.838484 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:44:42.839350 (+ 866us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:44:42.839811 (+ 461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:44:42.839817 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:44:42.839820 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:44:42.840154 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:44:42.840895 (+ 741us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:44:42.840901 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:44:42.841409 (+ 508us) client_negotiation.cc:769] Sending connection context
0926 03:44:42.841757 (+ 348us) client_negotiation.cc:240] Negotiation successful
0926 03:44:42.842309 (+ 552us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3283,"thread_start_us":3063,"threads_started":1}
I20260926 03:44:42.869154 8462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:42.854903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.193:40005 (local address 127.0.194.254:34071)
0926 03:44:42.855100 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0926 03:44:42.855106 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:44:42.855255 (+ 149us) server_negotiation.cc:407] Connection header received
0926 03:44:42.855474 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:44:42.855481 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:44:42.855562 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:44:42.855665 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:44:42.856587 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:42.857294 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:42.858615 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:42.859376 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:42.859523 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:42.861743 (+ 2220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:44:42.861752 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:44:42.861758 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:44:42.861806 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:44:42.865925 (+ 4119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:42.866803 (+ 878us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:42.866812 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:42.866819 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:42.866996 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:42.867460 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:42.867476 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:42.867482 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:42.867837 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:44:42.867968 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0926 03:44:42.868418 (+ 450us) server_negotiation.cc:299] Negotiation successful
0926 03:44:42.868678 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260926 03:44:42.869246 8544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:42.854322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:34071 (local address 127.0.194.193:40005)
0926 03:44:42.855145 (+ 823us) negotiation.cc:107] Waiting for socket to connect
0926 03:44:42.855168 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 03:44:42.855337 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:44:42.855688 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:44:42.855692 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:44:42.856157 (+ 465us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:44:42.856444 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:42.856453 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:42.857437 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:42.857443 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:42.858464 (+ 1021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:42.858474 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:42.859563 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:42.859570 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:42.859791 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:42.860625 (+ 834us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:44:42.860660 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:44:42.861482 (+ 822us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:44:42.866071 (+ 4589us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:44:42.866079 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:44:42.866088 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:44:42.866643 (+ 555us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:44:42.867114 (+ 471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:44:42.867120 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:44:42.867125 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:44:42.867305 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:44:42.867961 (+ 656us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:44:42.867965 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:44:42.868160 (+ 195us) client_negotiation.cc:769] Sending connection context
0926 03:44:42.868274 (+ 114us) client_negotiation.cc:240] Negotiation successful
0926 03:44:42.868695 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":646,"thread_start_us":450,"threads_started":1}
I20260926 03:44:42.870047 8438 rpc_server.cc:307] RPC server started. Bound to: 127.0.194.193:40371
I20260926 03:44:42.870095 8573 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.194.193:40371 every 8 connection(s)
I20260926 03:44:42.871385 8438 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790394221564982-779-0/minicluster-data/ts-0/data/info.pb
I20260926 03:44:42.873943 8574 heartbeater.cc:347] Connected to a master server at 127.0.194.254:34071
I20260926 03:44:42.874163 8574 heartbeater.cc:464] Registering TS with master...
I20260926 03:44:42.874850 8574 heartbeater.cc:511] Master 127.0.194.254:34071 requested a full tablet report, sending...
I20260926 03:44:42.876644 8367 ts_manager.cc:194] Registered new tserver with Master: ed1e5938558345edaccaa79fd9ed26b9 (127.0.194.193:40371)
I20260926 03:44:42.878069 8367 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.194.193@KRBTEST.COM'} at 127.0.194.193:36155
I20260926 03:44:42.879040 779 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu as pid 8438
I20260926 03:44:42.879243 779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790394221564982-779-0/minicluster-data/ts-0/wal/instance
I20260926 03:44:42.879866 779 external_mini_cluster.cc:1428] Setting key 9f6042bb751b34f3ed909dcab1ad18ac
WARNING: no policy specified for oryx/127.0.194.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.194.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.194.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790394221564982-779-0/krb5kdc/oryx_127.0.194.194.keytab.
Entry for principal oryx/127.0.194.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790394221564982-779-0/krb5kdc/oryx_127.0.194.194.keytab.
I20260926 03:44:42.927416 779 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
/tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790394221564982-779-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790394221564982-779-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790394221564982-779-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790394221564982-779-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.194.194:0
--local_ip_for_outbound_sockets=127.0.194.194
--webserver_interface=127.0.194.194
--webserver_port=0
--tserver_master_addrs=127.0.194.254:34071
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790394221564982-779-0/krb5kdc/oryx_127.0.194.194.keytab
--principal=oryx/127.0.194.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.194.212:41615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790394221564982-779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790394221564982-779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790394221564982-779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:44:43.253180 8578 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:44:43.253463 8578 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:44:43.253633 8578 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:44:43.260951 8578 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:44:43.261062 8578 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:44:43.261147 8578 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 03:44:43.261300 8578 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.194.194
I20260926 03:44:43.271176 8578 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.194.212:41615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790394221564982-779-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790394221564982-779-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790394221564982-779-0/krb5kdc/oryx_127.0.194.194.keytab
--principal=oryx/127.0.194.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.194.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790394221564982-779-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.194.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.194.254:34071
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.194.194
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790394221564982-779-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:44:43.272495 8578 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:44:43.273857 8578 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:44:43.286662 8583 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:44:43.286691 8586 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:44:43.286876 8584 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:44:43.287638 8578 server_base.cc:1034] running on GCE node
I20260926 03:44:43.288166 8578 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:44:43.289000 8578 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:44:43.290256 8578 hybrid_clock.cc:648] HybridClock initialized: now 1790394283290187 us; error 49 us; skew 500 ppm
I20260926 03:44:43.290584 8578 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:44:43 dist-test-slave-n2th krb5kdc[8322](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394283, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.194.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:44:43.293680 8578 init.cc:377] Logged in from keytab as oryx/127.0.194.194@KRBTEST.COM (short username oryx)
I20260926 03:44:43.295145 8578 webserver.cc:460] Webserver started at http://127.0.194.194:38927/ using document root <none> and password file <none>
I20260926 03:44:43.295784 8578 fs_manager.cc:359] Metadata directory not provided
I20260926 03:44:43.295867 8578 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:44:43.296128 8578 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:44:43.298962 8578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790394221564982-779-0/minicluster-data/ts-1/data/instance:
uuid: "ae87bdf37fd34fd5b5dc9416b1878954"
format_stamp: "Formatted at 2026-09-26 03:44:43 on dist-test-slave-n2th"
server_key: "20f820b09c00ee32359709175c387891"
server_key_iv: "eaba6589db24f76c3099bc47287015da"
server_key_version: "encryptionkey@0"
I20260926 03:44:43.299698 8578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790394221564982-779-0/minicluster-data/ts-1/wal/instance:
uuid: "ae87bdf37fd34fd5b5dc9416b1878954"
format_stamp: "Formatted at 2026-09-26 03:44:43 on dist-test-slave-n2th"
server_key: "20f820b09c00ee32359709175c387891"
server_key_iv: "eaba6589db24f76c3099bc47287015da"
server_key_version: "encryptionkey@0"
I20260926 03:44:43.304715 8578 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 03:44:43.308120 8593 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:44:43.309491 8578 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 03:44:43.309676 8578 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790394221564982-779-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790394221564982-779-0/minicluster-data/ts-1/wal
uuid: "ae87bdf37fd34fd5b5dc9416b1878954"
format_stamp: "Formatted at 2026-09-26 03:44:43 on dist-test-slave-n2th"
server_key: "20f820b09c00ee32359709175c387891"
server_key_iv: "eaba6589db24f76c3099bc47287015da"
server_key_version: "encryptionkey@0"
I20260926 03:44:43.309893 8578 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790394221564982-779-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790394221564982-779-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790394221564982-779-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:44:43.333189 8578 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:44:43.337131 8578 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:44:43.337409 8578 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:44:43.340236 8578 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:44:43.340342 8578 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:44:43.340538 8578 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:44:43.340646 8578 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 03:44:43 dist-test-slave-n2th krb5kdc[8322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394283, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.194.194@KRBTEST.COM for oryx/127.0.194.254@KRBTEST.COM
I20260926 03:44:43.363822 8462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:43.340665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.194:52655 (local address 127.0.194.254:34071)
0926 03:44:43.340859 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0926 03:44:43.340865 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:44:43.342235 (+ 1370us) server_negotiation.cc:407] Connection header received
0926 03:44:43.344383 (+ 2148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:44:43.344389 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:44:43.344507 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:44:43.344622 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:44:43.347046 (+ 2424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:43.348116 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:43.350986 (+ 2870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:43.351429 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:43.351575 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:43.356082 (+ 4507us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:44:43.356089 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:44:43.356094 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:44:43.356147 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:44:43.359613 (+ 3466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:43.360750 (+ 1137us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:43.360759 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:43.360765 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:43.360928 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:43.361402 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:43.361410 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:43.361415 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:43.361843 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:44:43.362023 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0926 03:44:43.363235 (+ 1212us) server_negotiation.cc:299] Negotiation successful
0926 03:44:43.363442 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260926 03:44:43.364877 8602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:43.341348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:34071 (local address 127.0.194.194:52655)
0926 03:44:43.342071 (+ 723us) negotiation.cc:107] Waiting for socket to connect
0926 03:44:43.342107 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 03:44:43.344038 (+ 1931us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:44:43.344864 (+ 826us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:44:43.344872 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:44:43.345745 (+ 873us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:44:43.346775 (+ 1030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:43.346810 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:43.348253 (+ 1443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:43.348258 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:43.350806 (+ 2548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:43.350823 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:43.351946 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:43.351952 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:43.352125 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:43.352829 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:44:43.352851 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:44:43.355799 (+ 2948us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:44:43.359817 (+ 4018us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:44:43.359827 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:44:43.359847 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:44:43.360587 (+ 740us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:44:43.361052 (+ 465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:44:43.361063 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:44:43.361068 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:44:43.361276 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:44:43.362122 (+ 846us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:44:43.362134 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:44:43.362687 (+ 553us) client_negotiation.cc:769] Sending connection context
0926 03:44:43.363142 (+ 455us) client_negotiation.cc:240] Negotiation successful
0926 03:44:43.363667 (+ 525us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":306,"threads_started":1}
I20260926 03:44:43.390830 8671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:43.374458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:34071 (local address 127.0.194.194:52157)
0926 03:44:43.375108 (+ 650us) negotiation.cc:107] Waiting for socket to connect
0926 03:44:43.375127 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 03:44:43.375291 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:44:43.375788 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:44:43.375797 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:44:43.376584 (+ 787us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:44:43.377069 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:43.377086 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:43.378372 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:43.378377 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:43.380664 (+ 2287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:43.380684 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:43.381680 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:43.381692 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:43.381893 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:43.382648 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:44:43.382677 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:44:43.383591 (+ 914us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:44:43.387868 (+ 4277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:44:43.387873 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:44:43.387878 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:44:43.388534 (+ 656us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:44:43.389004 (+ 470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:44:43.389007 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:44:43.389009 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:44:43.389123 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:44:43.389915 (+ 792us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:44:43.389918 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:44:43.390073 (+ 155us) client_negotiation.cc:769] Sending connection context
0926 03:44:43.390187 (+ 114us) client_negotiation.cc:240] Negotiation successful
0926 03:44:43.390416 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":532,"thread_start_us":387,"threads_started":1}
I20260926 03:44:43.391075 8462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:43.374066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.194:52157 (local address 127.0.194.254:34071)
0926 03:44:43.374265 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0926 03:44:43.374269 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:44:43.375214 (+ 945us) server_negotiation.cc:407] Connection header received
0926 03:44:43.375508 (+ 294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:44:43.375516 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:44:43.375616 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:44:43.375727 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:44:43.377272 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:43.378207 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:43.380829 (+ 2622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:43.381473 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:43.381623 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:43.383780 (+ 2157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:44:43.383786 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:44:43.383791 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:44:43.383839 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:44:43.387675 (+ 3836us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:43.388677 (+ 1002us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:43.388686 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:43.388693 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:43.388880 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:43.389239 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:43.389247 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:43.389253 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:43.389788 (+ 535us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:44:43.389972 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0926 03:44:43.390349 (+ 377us) server_negotiation.cc:299] Negotiation successful
0926 03:44:43.390628 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260926 03:44:43.394042 8578 rpc_server.cc:307] RPC server started. Bound to: 127.0.194.194:34427
I20260926 03:44:43.394069 8712 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.194.194:34427 every 8 connection(s)
I20260926 03:44:43.395471 8578 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790394221564982-779-0/minicluster-data/ts-1/data/info.pb
I20260926 03:44:43.397261 8713 heartbeater.cc:347] Connected to a master server at 127.0.194.254:34071
I20260926 03:44:43.397440 8713 heartbeater.cc:464] Registering TS with master...
I20260926 03:44:43.397967 8713 heartbeater.cc:511] Master 127.0.194.254:34071 requested a full tablet report, sending...
I20260926 03:44:43.398989 779 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu as pid 8578
I20260926 03:44:43.399160 779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790394221564982-779-0/minicluster-data/ts-1/wal/instance
I20260926 03:44:43.399219 8367 ts_manager.cc:194] Registered new tserver with Master: ae87bdf37fd34fd5b5dc9416b1878954 (127.0.194.194:34427)
I20260926 03:44:43.399650 779 external_mini_cluster.cc:1428] Setting key 0ad20a9ab62ac4181fbd233d761252bb
I20260926 03:44:43.400079 8367 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.194.194@KRBTEST.COM'} at 127.0.194.194:52655
WARNING: no policy specified for oryx/127.0.194.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.194.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.194.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790394221564982-779-0/krb5kdc/oryx_127.0.194.195.keytab.
Entry for principal oryx/127.0.194.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790394221564982-779-0/krb5kdc/oryx_127.0.194.195.keytab.
I20260926 03:44:43.443939 779 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
/tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790394221564982-779-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790394221564982-779-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790394221564982-779-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790394221564982-779-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.194.195:0
--local_ip_for_outbound_sockets=127.0.194.195
--webserver_interface=127.0.194.195
--webserver_port=0
--tserver_master_addrs=127.0.194.254:34071
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790394221564982-779-0/krb5kdc/oryx_127.0.194.195.keytab
--principal=oryx/127.0.194.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.194.212:41615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790394221564982-779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790394221564982-779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790394221564982-779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:44:43.774603 8717 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:44:43.774971 8717 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:44:43.775166 8717 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:44:43.782626 8717 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:44:43.782755 8717 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:44:43.782907 8717 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 03:44:43.783035 8717 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.194.195
I20260926 03:44:43.793030 8717 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.194.212:41615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790394221564982-779-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790394221564982-779-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790394221564982-779-0/krb5kdc/oryx_127.0.194.195.keytab
--principal=oryx/127.0.194.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.194.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790394221564982-779-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.194.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.194.254:34071
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.194.195
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790394221564982-779-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:44:43.794330 8717 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:44:43.795684 8717 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:44:43.808677 8725 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:44:43.808682 8722 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:44:43.808629 8723 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:44:43.809346 8717 server_base.cc:1034] running on GCE node
I20260926 03:44:43.809890 8717 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:44:43.810719 8717 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:44:43.811998 8717 hybrid_clock.cc:648] HybridClock initialized: now 1790394283811941 us; error 38 us; skew 500 ppm
I20260926 03:44:43.812300 8717 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:44:43 dist-test-slave-n2th krb5kdc[8322](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394283, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.194.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:44:43.815475 8717 init.cc:377] Logged in from keytab as oryx/127.0.194.195@KRBTEST.COM (short username oryx)
I20260926 03:44:43.816831 8717 webserver.cc:460] Webserver started at http://127.0.194.195:39347/ using document root <none> and password file <none>
I20260926 03:44:43.817507 8717 fs_manager.cc:359] Metadata directory not provided
I20260926 03:44:43.817642 8717 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:44:43.817965 8717 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:44:43.820890 8717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790394221564982-779-0/minicluster-data/ts-2/data/instance:
uuid: "8b25e3df61ce40b1818bdb428b3d47a1"
format_stamp: "Formatted at 2026-09-26 03:44:43 on dist-test-slave-n2th"
server_key: "b50ec4f9c91b3c68bbbbf003d6e3ec98"
server_key_iv: "aa6c45053ebaf10393888cb923a41800"
server_key_version: "encryptionkey@0"
I20260926 03:44:43.821698 8717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790394221564982-779-0/minicluster-data/ts-2/wal/instance:
uuid: "8b25e3df61ce40b1818bdb428b3d47a1"
format_stamp: "Formatted at 2026-09-26 03:44:43 on dist-test-slave-n2th"
server_key: "b50ec4f9c91b3c68bbbbf003d6e3ec98"
server_key_iv: "aa6c45053ebaf10393888cb923a41800"
server_key_version: "encryptionkey@0"
I20260926 03:44:43.827229 8717 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260926 03:44:43.830691 8732 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:44:43.832101 8717 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 03:44:43.832288 8717 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790394221564982-779-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790394221564982-779-0/minicluster-data/ts-2/wal
uuid: "8b25e3df61ce40b1818bdb428b3d47a1"
format_stamp: "Formatted at 2026-09-26 03:44:43 on dist-test-slave-n2th"
server_key: "b50ec4f9c91b3c68bbbbf003d6e3ec98"
server_key_iv: "aa6c45053ebaf10393888cb923a41800"
server_key_version: "encryptionkey@0"
I20260926 03:44:43.832536 8717 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790394221564982-779-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790394221564982-779-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790394221564982-779-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:44:43.858390 8717 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:44:43.862592 8717 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:44:43.862932 8717 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:44:43.865801 8717 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:44:43.865914 8717 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:44:43.866078 8717 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:44:43.866155 8717 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 03:44:43 dist-test-slave-n2th krb5kdc[8322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394283, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.194.195@KRBTEST.COM for oryx/127.0.194.254@KRBTEST.COM
I20260926 03:44:43.881412 8574 heartbeater.cc:503] Master 127.0.194.254:34071 was elected leader, sending a full tablet report...
I20260926 03:44:43.891573 8462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:43.866989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.195:58141 (local address 127.0.194.254:34071)
0926 03:44:43.867757 (+ 768us) server_negotiation.cc:206] Beginning negotiation
0926 03:44:43.867765 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:44:43.868411 (+ 646us) server_negotiation.cc:407] Connection header received
0926 03:44:43.871286 (+ 2875us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:44:43.871296 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:44:43.871409 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:44:43.871531 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:44:43.873488 (+ 1957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:43.875037 (+ 1549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:43.876670 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:43.877574 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:43.877735 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:43.882269 (+ 4534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:44:43.882279 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:44:43.882287 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:44:43.882340 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:44:43.886749 (+ 4409us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:43.888042 (+ 1293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:43.888050 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:43.888057 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:43.888246 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:43.888928 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:43.888933 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:43.888938 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:43.889400 (+ 462us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:44:43.889590 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0926 03:44:43.890953 (+ 1363us) server_negotiation.cc:299] Negotiation successful
0926 03:44:43.891179 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":613}
I20260926 03:44:43.892261 8741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:43.867325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:34071 (local address 127.0.194.195:58141)
0926 03:44:43.868230 (+ 905us) negotiation.cc:107] Waiting for socket to connect
0926 03:44:43.868281 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 03:44:43.870916 (+ 2635us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:44:43.871791 (+ 875us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:44:43.871802 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:44:43.872521 (+ 719us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:44:43.873264 (+ 743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:43.873290 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:43.875217 (+ 1927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:43.875224 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:43.876526 (+ 1302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:43.876537 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:43.877785 (+ 1248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:43.877790 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:43.877965 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:43.878692 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:44:43.878733 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:44:43.882009 (+ 3276us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:44:43.886925 (+ 4916us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:44:43.886938 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:44:43.886961 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:44:43.887842 (+ 881us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:44:43.888406 (+ 564us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:44:43.888414 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:44:43.888420 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:44:43.888787 (+ 367us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:44:43.889540 (+ 753us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:44:43.889546 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:44:43.889889 (+ 343us) client_negotiation.cc:769] Sending connection context
0926 03:44:43.890314 (+ 425us) client_negotiation.cc:240] Negotiation successful
0926 03:44:43.890783 (+ 469us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":574,"thread_start_us":375,"threads_started":1}
I20260926 03:44:43.925098 8462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:43.906148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.195:47629 (local address 127.0.194.254:34071)
0926 03:44:43.906364 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0926 03:44:43.906373 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:44:43.907821 (+ 1448us) server_negotiation.cc:407] Connection header received
0926 03:44:43.907987 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:44:43.907996 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:44:43.908105 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:44:43.908234 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:44:43.909997 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:43.911277 (+ 1280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:43.913968 (+ 2691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:43.914476 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:43.914655 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:43.917648 (+ 2993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:44:43.917657 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:44:43.917664 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:44:43.917708 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:44:43.921562 (+ 3854us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:43.922602 (+ 1040us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:43.922609 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:43.922612 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:43.922803 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:43.923316 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:43.923323 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:43.923327 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:43.923604 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:44:43.923720 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0926 03:44:43.924399 (+ 679us) server_negotiation.cc:299] Negotiation successful
0926 03:44:43.924687 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 03:44:43.925237 8794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:43.906097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:34071 (local address 127.0.194.195:47629)
0926 03:44:43.907693 (+ 1596us) negotiation.cc:107] Waiting for socket to connect
0926 03:44:43.907713 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 03:44:43.907849 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:44:43.908440 (+ 591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:44:43.908446 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:44:43.909251 (+ 805us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:44:43.909800 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:43.909820 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:43.911446 (+ 1626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:43.911452 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:43.913789 (+ 2337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:43.913806 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:43.914727 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:43.914755 (+ 28us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:43.915002 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:43.915813 (+ 811us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:44:43.915846 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:44:43.917405 (+ 1559us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:44:43.921721 (+ 4316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:44:43.921732 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:44:43.921739 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:44:43.922449 (+ 710us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:44:43.922960 (+ 511us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:44:43.922977 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:44:43.922983 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:44:43.923183 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:44:43.923742 (+ 559us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:44:43.923749 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:44:43.924031 (+ 282us) client_negotiation.cc:769] Sending connection context
0926 03:44:43.924243 (+ 212us) client_negotiation.cc:240] Negotiation successful
0926 03:44:43.924726 (+ 483us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1442,"thread_start_us":1254,"threads_started":1}
I20260926 03:44:43.926857 8717 rpc_server.cc:307] RPC server started. Bound to: 127.0.194.195:45461
I20260926 03:44:43.926908 8851 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.194.195:45461 every 8 connection(s)
I20260926 03:44:43.928156 8717 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790394221564982-779-0/minicluster-data/ts-2/data/info.pb
I20260926 03:44:43.930348 8852 heartbeater.cc:347] Connected to a master server at 127.0.194.254:34071
I20260926 03:44:43.930606 8852 heartbeater.cc:464] Registering TS with master...
I20260926 03:44:43.931259 8852 heartbeater.cc:511] Master 127.0.194.254:34071 requested a full tablet report, sending...
I20260926 03:44:43.932561 8367 ts_manager.cc:194] Registered new tserver with Master: 8b25e3df61ce40b1818bdb428b3d47a1 (127.0.194.195:45461)
I20260926 03:44:43.933349 8367 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.194.195@KRBTEST.COM'} at 127.0.194.195:58141
I20260926 03:44:43.935242 779 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu as pid 8717
I20260926 03:44:43.935415 779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790394221564982-779-0/minicluster-data/ts-2/wal/instance
I20260926 03:44:43.935895 779 external_mini_cluster.cc:1428] Setting key 9f24eed3e33116429191da29fcc9c6b2
I20260926 03:44:43.938668 779 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 03:44:43 dist-test-slave-n2th krb5kdc[8322](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394283, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 03:44:43 dist-test-slave-n2th krb5kdc[8322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.194.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:44:43 dist-test-slave-n2th krb5kdc[8322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.194.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:44:44.403066 8713 heartbeater.cc:503] Master 127.0.194.254:34071 was elected leader, sending a full tablet report...
I20260926 03:44:44.936028 8852 heartbeater.cc:503] Master 127.0.194.254:34071 was elected leader, sending a full tablet report...
W20260926 03:44:45.040766 8860 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.058s user 0.000s sys 0.014s
W20260926 03:44:45.041946 8462 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:44:43.977129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56322 (local address 127.0.194.254:34071)
0926 03:44:43.977352 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0926 03:44:43.977360 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:44:43.977498 (+ 138us) server_negotiation.cc:407] Connection header received
0926 03:44:43.977646 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:44:43.977651 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:44:43.977714 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:44:43.977798 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:44:43.979095 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:43.979826 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:43.981124 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:43.981616 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:43.981707 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:45.041695 (+1059988us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56322: BlockingRecv error: recv got EOF from 127.0.0.1:56322 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Sep 26 03:44:45 dist-test-slave-n2th krb5kdc[8322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394283, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.194.254@KRBTEST.COM
I20260926 03:44:45.066777 8462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:45.046719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56336 (local address 127.0.194.254:34071)
0926 03:44:45.046976 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0926 03:44:45.046982 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:44:45.047301 (+ 319us) server_negotiation.cc:407] Connection header received
0926 03:44:45.047457 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:44:45.047461 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:44:45.047545 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:44:45.047634 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:44:45.048832 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:45.049577 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:45.050863 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:45.051254 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:45.051351 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:45.061037 (+ 9686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:44:45.061044 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:44:45.061048 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:44:45.061108 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:44:45.063876 (+ 2768us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:45.064697 (+ 821us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:45.064702 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:45.064704 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:45.064842 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:45.065252 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:45.065256 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:45.065258 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:45.065585 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:44:45.065890 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0926 03:44:45.066175 (+ 285us) server_negotiation.cc:299] Negotiation successful
0926 03:44:45.066423 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260926 03:44:45.071085 8367 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56336:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 03:44:45.079068 8367 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 03:44:45.128044 8873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:45.091962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.194:34427 (local address 127.0.0.1:45630)
0926 03:44:45.115915 (+ 23953us) negotiation.cc:107] Waiting for socket to connect
0926 03:44:45.115939 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 03:44:45.116262 (+ 323us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:44:45.117565 (+ 1303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:44:45.117575 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:44:45.117610 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:44:45.118551 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:45.118568 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:45.120877 (+ 2309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:45.120882 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:45.124904 (+ 4022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:45.124922 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:45.126959 (+ 2037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:45.126966 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:45.127166 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:45.127221 (+ 55us) client_negotiation.cc:769] Sending connection context
0926 03:44:45.127384 (+ 163us) client_negotiation.cc:240] Negotiation successful
0926 03:44:45.127604 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23715,"mutex_wait_us":33,"thread_start_us":24787,"threads_started":1}
I20260926 03:44:45.128669 8872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:45.091625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47720 (local address 127.0.194.195:45461)
0926 03:44:45.107191 (+ 15566us) server_negotiation.cc:206] Beginning negotiation
0926 03:44:45.107207 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 03:44:45.116093 (+ 8886us) server_negotiation.cc:407] Connection header received
0926 03:44:45.116949 (+ 856us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:44:45.116959 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:44:45.117200 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:44:45.117378 (+ 178us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:44:45.118589 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:45.120078 (+ 1489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:45.125608 (+ 5530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:45.127326 (+ 1718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:45.127453 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:45.127651 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0926 03:44:45.128226 (+ 575us) server_negotiation.cc:299] Negotiation successful
0926 03:44:45.128396 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15289,"thread_start_us":15583,"threads_started":1}
I20260926 03:44:45.129016 8874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:45.092170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45630 (local address 127.0.194.194:34427)
0926 03:44:45.115180 (+ 23010us) server_negotiation.cc:206] Beginning negotiation
0926 03:44:45.115191 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 03:44:45.116073 (+ 882us) server_negotiation.cc:407] Connection header received
0926 03:44:45.116559 (+ 486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:44:45.116571 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:44:45.117033 (+ 462us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:44:45.117206 (+ 173us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:44:45.118857 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:45.120693 (+ 1836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:45.125068 (+ 4375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:45.126409 (+ 1341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:45.126569 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:45.126823 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0926 03:44:45.128451 (+ 1628us) server_negotiation.cc:299] Negotiation successful
0926 03:44:45.128626 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22812,"thread_start_us":28548,"threads_started":1}
I20260926 03:44:45.130113 8876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:45.094985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42844 (local address 127.0.194.193:40371)
0926 03:44:45.116552 (+ 21567us) server_negotiation.cc:206] Beginning negotiation
0926 03:44:45.116564 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 03:44:45.117781 (+ 1217us) server_negotiation.cc:407] Connection header received
0926 03:44:45.117901 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:44:45.117909 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:44:45.118137 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:44:45.118302 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:44:45.119186 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:45.121870 (+ 2684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:45.125710 (+ 3840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:45.127629 (+ 1919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:45.127796 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:45.128023 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0926 03:44:45.129548 (+ 1525us) server_negotiation.cc:299] Negotiation successful
0926 03:44:45.129769 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21406,"thread_start_us":21886,"threads_started":1}
I20260926 03:44:45.129292 8871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:45.094344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.193:40371 (local address 127.0.0.1:42844)
0926 03:44:45.117464 (+ 23120us) negotiation.cc:107] Waiting for socket to connect
0926 03:44:45.117481 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 03:44:45.117594 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:44:45.118499 (+ 905us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:44:45.118503 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:44:45.118529 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:44:45.118995 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:45.119008 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:45.122057 (+ 3049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:45.122063 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:45.125532 (+ 3469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:45.125541 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:45.128188 (+ 2647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:45.128197 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:45.128398 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:45.128442 (+ 44us) client_negotiation.cc:769] Sending connection context
0926 03:44:45.128531 (+ 89us) client_negotiation.cc:240] Negotiation successful
0926 03:44:45.128689 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22997,"mutex_wait_us":3,"thread_start_us":20302,"threads_started":1}
I20260926 03:44:45.129745 8875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:45.091539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.195:45461 (local address 127.0.0.1:47720)
0926 03:44:45.115884 (+ 24345us) negotiation.cc:107] Waiting for socket to connect
0926 03:44:45.115932 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 03:44:45.116737 (+ 805us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:44:45.117637 (+ 900us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:44:45.117643 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:44:45.117672 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:44:45.118153 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:45.118170 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:45.120262 (+ 2092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:45.120267 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:45.125424 (+ 5157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:45.125438 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:45.127802 (+ 2364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:45.127808 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:45.127964 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:45.128006 (+ 42us) client_negotiation.cc:769] Sending connection context
0926 03:44:45.129213 (+ 1207us) client_negotiation.cc:240] Negotiation successful
0926 03:44:45.129366 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22797,"mutex_wait_us":106,"thread_start_us":29605,"threads_started":1}
I20260926 03:44:45.134215 8786 tablet_service.cc:1467] Processing CreateTablet for tablet 6c0757c73551430b98c92b68aa0b2c05 (DEFAULT_TABLE table=test-table [id=15f0d2c0750b4aa483d63d93c3b997cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:44:45.134456 8504 tablet_service.cc:1467] Processing CreateTablet for tablet 6c0757c73551430b98c92b68aa0b2c05 (DEFAULT_TABLE table=test-table [id=15f0d2c0750b4aa483d63d93c3b997cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:44:45.135876 8504 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c0757c73551430b98c92b68aa0b2c05. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:44:45.136004 8786 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c0757c73551430b98c92b68aa0b2c05. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:44:45.138700 8643 tablet_service.cc:1467] Processing CreateTablet for tablet 6c0757c73551430b98c92b68aa0b2c05 (DEFAULT_TABLE table=test-table [id=15f0d2c0750b4aa483d63d93c3b997cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:44:45.140206 8643 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c0757c73551430b98c92b68aa0b2c05. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:44:45.150481 8877 tablet_bootstrap.cc:492] T 6c0757c73551430b98c92b68aa0b2c05 P 8b25e3df61ce40b1818bdb428b3d47a1: Bootstrap starting.
I20260926 03:44:45.154331 8878 tablet_bootstrap.cc:492] T 6c0757c73551430b98c92b68aa0b2c05 P ed1e5938558345edaccaa79fd9ed26b9: Bootstrap starting.
I20260926 03:44:45.155479 8877 tablet_bootstrap.cc:654] T 6c0757c73551430b98c92b68aa0b2c05 P 8b25e3df61ce40b1818bdb428b3d47a1: Neither blocks nor log segments found. Creating new log.
I20260926 03:44:45.155725 8879 tablet_bootstrap.cc:492] T 6c0757c73551430b98c92b68aa0b2c05 P ae87bdf37fd34fd5b5dc9416b1878954: Bootstrap starting.
I20260926 03:44:45.157050 8877 log.cc:824] T 6c0757c73551430b98c92b68aa0b2c05 P 8b25e3df61ce40b1818bdb428b3d47a1: Log is configured to *not* fsync() on all Append() calls
I20260926 03:44:45.158017 8878 tablet_bootstrap.cc:654] T 6c0757c73551430b98c92b68aa0b2c05 P ed1e5938558345edaccaa79fd9ed26b9: Neither blocks nor log segments found. Creating new log.
I20260926 03:44:45.159404 8879 tablet_bootstrap.cc:654] T 6c0757c73551430b98c92b68aa0b2c05 P ae87bdf37fd34fd5b5dc9416b1878954: Neither blocks nor log segments found. Creating new log.
I20260926 03:44:45.159595 8878 log.cc:824] T 6c0757c73551430b98c92b68aa0b2c05 P ed1e5938558345edaccaa79fd9ed26b9: Log is configured to *not* fsync() on all Append() calls
I20260926 03:44:45.160691 8879 log.cc:824] T 6c0757c73551430b98c92b68aa0b2c05 P ae87bdf37fd34fd5b5dc9416b1878954: Log is configured to *not* fsync() on all Append() calls
I20260926 03:44:45.164601 8877 tablet_bootstrap.cc:492] T 6c0757c73551430b98c92b68aa0b2c05 P 8b25e3df61ce40b1818bdb428b3d47a1: No bootstrap required, opened a new log
I20260926 03:44:45.165050 8877 ts_tablet_manager.cc:1397] T 6c0757c73551430b98c92b68aa0b2c05 P 8b25e3df61ce40b1818bdb428b3d47a1: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.003s
I20260926 03:44:45.169437 8878 tablet_bootstrap.cc:492] T 6c0757c73551430b98c92b68aa0b2c05 P ed1e5938558345edaccaa79fd9ed26b9: No bootstrap required, opened a new log
I20260926 03:44:45.169739 8878 ts_tablet_manager.cc:1397] T 6c0757c73551430b98c92b68aa0b2c05 P ed1e5938558345edaccaa79fd9ed26b9: Time spent bootstrapping tablet: real 0.016s user 0.009s sys 0.005s
I20260926 03:44:45.176716 8879 tablet_bootstrap.cc:492] T 6c0757c73551430b98c92b68aa0b2c05 P ae87bdf37fd34fd5b5dc9416b1878954: No bootstrap required, opened a new log
I20260926 03:44:45.177135 8879 ts_tablet_manager.cc:1397] T 6c0757c73551430b98c92b68aa0b2c05 P ae87bdf37fd34fd5b5dc9416b1878954: Time spent bootstrapping tablet: real 0.022s user 0.009s sys 0.009s
I20260926 03:44:45.178004 8877 raft_consensus.cc:348] T 6c0757c73551430b98c92b68aa0b2c05 P 8b25e3df61ce40b1818bdb428b3d47a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b25e3df61ce40b1818bdb428b3d47a1" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 45461 } } peers { permanent_uuid: "ae87bdf37fd34fd5b5dc9416b1878954" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 34427 } } peers { permanent_uuid: "ed1e5938558345edaccaa79fd9ed26b9" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 40371 } }
I20260926 03:44:45.178462 8877 raft_consensus.cc:374] T 6c0757c73551430b98c92b68aa0b2c05 P 8b25e3df61ce40b1818bdb428b3d47a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:44:45.178568 8877 raft_consensus.cc:729] T 6c0757c73551430b98c92b68aa0b2c05 P 8b25e3df61ce40b1818bdb428b3d47a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b25e3df61ce40b1818bdb428b3d47a1, State: Initialized, Role: FOLLOWER
I20260926 03:44:45.179301 8877 consensus_queue.cc:260] T 6c0757c73551430b98c92b68aa0b2c05 P 8b25e3df61ce40b1818bdb428b3d47a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b25e3df61ce40b1818bdb428b3d47a1" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 45461 } } peers { permanent_uuid: "ae87bdf37fd34fd5b5dc9416b1878954" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 34427 } } peers { permanent_uuid: "ed1e5938558345edaccaa79fd9ed26b9" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 40371 } }
I20260926 03:44:45.181226 8877 ts_tablet_manager.cc:1428] T 6c0757c73551430b98c92b68aa0b2c05 P 8b25e3df61ce40b1818bdb428b3d47a1: Time spent starting tablet: real 0.016s user 0.012s sys 0.004s
I20260926 03:44:45.181665 8878 raft_consensus.cc:348] T 6c0757c73551430b98c92b68aa0b2c05 P ed1e5938558345edaccaa79fd9ed26b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b25e3df61ce40b1818bdb428b3d47a1" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 45461 } } peers { permanent_uuid: "ae87bdf37fd34fd5b5dc9416b1878954" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 34427 } } peers { permanent_uuid: "ed1e5938558345edaccaa79fd9ed26b9" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 40371 } }
I20260926 03:44:45.182116 8878 raft_consensus.cc:374] T 6c0757c73551430b98c92b68aa0b2c05 P ed1e5938558345edaccaa79fd9ed26b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:44:45.182228 8878 raft_consensus.cc:729] T 6c0757c73551430b98c92b68aa0b2c05 P ed1e5938558345edaccaa79fd9ed26b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed1e5938558345edaccaa79fd9ed26b9, State: Initialized, Role: FOLLOWER
I20260926 03:44:45.182898 8878 consensus_queue.cc:260] T 6c0757c73551430b98c92b68aa0b2c05 P ed1e5938558345edaccaa79fd9ed26b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b25e3df61ce40b1818bdb428b3d47a1" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 45461 } } peers { permanent_uuid: "ae87bdf37fd34fd5b5dc9416b1878954" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 34427 } } peers { permanent_uuid: "ed1e5938558345edaccaa79fd9ed26b9" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 40371 } }
I20260926 03:44:45.184672 8878 ts_tablet_manager.cc:1428] T 6c0757c73551430b98c92b68aa0b2c05 P ed1e5938558345edaccaa79fd9ed26b9: Time spent starting tablet: real 0.015s user 0.010s sys 0.003s
I20260926 03:44:45.190353 8879 raft_consensus.cc:348] T 6c0757c73551430b98c92b68aa0b2c05 P ae87bdf37fd34fd5b5dc9416b1878954 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b25e3df61ce40b1818bdb428b3d47a1" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 45461 } } peers { permanent_uuid: "ae87bdf37fd34fd5b5dc9416b1878954" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 34427 } } peers { permanent_uuid: "ed1e5938558345edaccaa79fd9ed26b9" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 40371 } }
I20260926 03:44:45.190727 8879 raft_consensus.cc:374] T 6c0757c73551430b98c92b68aa0b2c05 P ae87bdf37fd34fd5b5dc9416b1878954 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:44:45.190855 8879 raft_consensus.cc:729] T 6c0757c73551430b98c92b68aa0b2c05 P ae87bdf37fd34fd5b5dc9416b1878954 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae87bdf37fd34fd5b5dc9416b1878954, State: Initialized, Role: FOLLOWER
I20260926 03:44:45.191411 8879 consensus_queue.cc:260] T 6c0757c73551430b98c92b68aa0b2c05 P ae87bdf37fd34fd5b5dc9416b1878954 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b25e3df61ce40b1818bdb428b3d47a1" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 45461 } } peers { permanent_uuid: "ae87bdf37fd34fd5b5dc9416b1878954" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 34427 } } peers { permanent_uuid: "ed1e5938558345edaccaa79fd9ed26b9" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 40371 } }
I20260926 03:44:45.192781 8879 ts_tablet_manager.cc:1428] T 6c0757c73551430b98c92b68aa0b2c05 P ae87bdf37fd34fd5b5dc9416b1878954: Time spent starting tablet: real 0.015s user 0.010s sys 0.002s
I20260926 03:44:45.202508 8884 raft_consensus.cc:482] T 6c0757c73551430b98c92b68aa0b2c05 P ed1e5938558345edaccaa79fd9ed26b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 03:44:45.202860 8884 raft_consensus.cc:504] T 6c0757c73551430b98c92b68aa0b2c05 P ed1e5938558345edaccaa79fd9ed26b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b25e3df61ce40b1818bdb428b3d47a1" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 45461 } } peers { permanent_uuid: "ae87bdf37fd34fd5b5dc9416b1878954" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 34427 } } peers { permanent_uuid: "ed1e5938558345edaccaa79fd9ed26b9" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 40371 } }
I20260926 03:44:45.204582 8884 leader_election.cc:288] T 6c0757c73551430b98c92b68aa0b2c05 P ed1e5938558345edaccaa79fd9ed26b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8b25e3df61ce40b1818bdb428b3d47a1 (127.0.194.195:45461), ae87bdf37fd34fd5b5dc9416b1878954 (127.0.194.194:34427)
I20260926 03:44:45.211894 8874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:45.205227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.193:57701 (local address 127.0.194.194:34427)
0926 03:44:45.205422 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0926 03:44:45.205429 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:44:45.206279 (+ 850us) server_negotiation.cc:407] Connection header received
0926 03:44:45.206535 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:44:45.206539 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:44:45.206617 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:44:45.206709 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:44:45.207458 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:45.208137 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:45.210003 (+ 1866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:45.210856 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:45.211012 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:45.211099 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0926 03:44:45.211539 (+ 440us) server_negotiation.cc:299] Negotiation successful
0926 03:44:45.211655 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 03:44:45.211906 8872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:45.205188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.193:41873 (local address 127.0.194.195:45461)
0926 03:44:45.205418 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0926 03:44:45.205428 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 03:44:45.206069 (+ 641us) server_negotiation.cc:407] Connection header received
0926 03:44:45.206351 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:44:45.206357 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:44:45.206458 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:44:45.206523 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:44:45.207375 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:45.208099 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:45.210051 (+ 1952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:45.210876 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:45.211062 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:45.211130 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 03:44:45.211511 (+ 381us) server_negotiation.cc:299] Negotiation successful
0926 03:44:45.211650 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 03:44:45.212035 8886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:45.204923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.195:45461 (local address 127.0.194.193:41873)
0926 03:44:45.205938 (+ 1015us) negotiation.cc:107] Waiting for socket to connect
0926 03:44:45.205953 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 03:44:45.206193 (+ 240us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:44:45.206723 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:44:45.206727 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:44:45.206770 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:44:45.207218 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:45.207231 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:45.208228 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:45.208235 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:45.209869 (+ 1634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:45.209880 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:45.211121 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:45.211128 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:45.211357 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:45.211381 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 03:44:45.211476 (+ 95us) client_negotiation.cc:240] Negotiation successful
0926 03:44:45.211610 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":889,"mutex_wait_us":106,"thread_start_us":695,"threads_started":1}
I20260926 03:44:45.212303 8887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:45.204927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.194:34427 (local address 127.0.194.193:57701)
0926 03:44:45.206160 (+ 1233us) negotiation.cc:107] Waiting for socket to connect
0926 03:44:45.206183 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 03:44:45.206383 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:44:45.206768 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:44:45.206776 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:44:45.206809 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:44:45.207305 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:45.207320 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:45.208294 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:45.208298 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:45.209847 (+ 1549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:45.209861 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:45.211073 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:45.211079 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:45.211365 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:45.211388 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 03:44:45.211493 (+ 105us) client_negotiation.cc:240] Negotiation successful
0926 03:44:45.211691 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1090,"mutex_wait_us":656,"thread_start_us":385,"threads_started":1}
I20260926 03:44:45.213066 8807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c0757c73551430b98c92b68aa0b2c05" candidate_uuid: "ed1e5938558345edaccaa79fd9ed26b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b25e3df61ce40b1818bdb428b3d47a1" is_pre_election: true
I20260926 03:44:45.213593 8807 raft_consensus.cc:2454] T 6c0757c73551430b98c92b68aa0b2c05 P 8b25e3df61ce40b1818bdb428b3d47a1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed1e5938558345edaccaa79fd9ed26b9 in term 0.
I20260926 03:44:45.213786 8664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c0757c73551430b98c92b68aa0b2c05" candidate_uuid: "ed1e5938558345edaccaa79fd9ed26b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae87bdf37fd34fd5b5dc9416b1878954" is_pre_election: true
I20260926 03:44:45.214304 8454 leader_election.cc:302] T 6c0757c73551430b98c92b68aa0b2c05 P ed1e5938558345edaccaa79fd9ed26b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b25e3df61ce40b1818bdb428b3d47a1, ed1e5938558345edaccaa79fd9ed26b9; no voters:
I20260926 03:44:45.214360 8664 raft_consensus.cc:2454] T 6c0757c73551430b98c92b68aa0b2c05 P ae87bdf37fd34fd5b5dc9416b1878954 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed1e5938558345edaccaa79fd9ed26b9 in term 0.
I20260926 03:44:45.214809 8884 raft_consensus.cc:2781] T 6c0757c73551430b98c92b68aa0b2c05 P ed1e5938558345edaccaa79fd9ed26b9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 03:44:45.214973 8884 raft_consensus.cc:482] T 6c0757c73551430b98c92b68aa0b2c05 P ed1e5938558345edaccaa79fd9ed26b9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 03:44:45.215102 8884 raft_consensus.cc:3037] T 6c0757c73551430b98c92b68aa0b2c05 P ed1e5938558345edaccaa79fd9ed26b9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:44:45.217221 8884 raft_consensus.cc:504] T 6c0757c73551430b98c92b68aa0b2c05 P ed1e5938558345edaccaa79fd9ed26b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b25e3df61ce40b1818bdb428b3d47a1" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 45461 } } peers { permanent_uuid: "ae87bdf37fd34fd5b5dc9416b1878954" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 34427 } } peers { permanent_uuid: "ed1e5938558345edaccaa79fd9ed26b9" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 40371 } }
I20260926 03:44:45.217814 8884 leader_election.cc:288] T 6c0757c73551430b98c92b68aa0b2c05 P ed1e5938558345edaccaa79fd9ed26b9 [CANDIDATE]: Term 1 election: Requested vote from peers 8b25e3df61ce40b1818bdb428b3d47a1 (127.0.194.195:45461), ae87bdf37fd34fd5b5dc9416b1878954 (127.0.194.194:34427)
I20260926 03:44:45.218231 8807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c0757c73551430b98c92b68aa0b2c05" candidate_uuid: "ed1e5938558345edaccaa79fd9ed26b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b25e3df61ce40b1818bdb428b3d47a1"
I20260926 03:44:45.218258 8664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c0757c73551430b98c92b68aa0b2c05" candidate_uuid: "ed1e5938558345edaccaa79fd9ed26b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae87bdf37fd34fd5b5dc9416b1878954"
I20260926 03:44:45.218438 8807 raft_consensus.cc:3037] T 6c0757c73551430b98c92b68aa0b2c05 P 8b25e3df61ce40b1818bdb428b3d47a1 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:44:45.218436 8664 raft_consensus.cc:3037] T 6c0757c73551430b98c92b68aa0b2c05 P ae87bdf37fd34fd5b5dc9416b1878954 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:44:45.220681 8664 raft_consensus.cc:2454] T 6c0757c73551430b98c92b68aa0b2c05 P ae87bdf37fd34fd5b5dc9416b1878954 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed1e5938558345edaccaa79fd9ed26b9 in term 1.
I20260926 03:44:45.221097 8456 leader_election.cc:302] T 6c0757c73551430b98c92b68aa0b2c05 P ed1e5938558345edaccaa79fd9ed26b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae87bdf37fd34fd5b5dc9416b1878954, ed1e5938558345edaccaa79fd9ed26b9; no voters:
I20260926 03:44:45.221243 8807 raft_consensus.cc:2454] T 6c0757c73551430b98c92b68aa0b2c05 P 8b25e3df61ce40b1818bdb428b3d47a1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed1e5938558345edaccaa79fd9ed26b9 in term 1.
I20260926 03:44:45.221374 8884 raft_consensus.cc:2781] T 6c0757c73551430b98c92b68aa0b2c05 P ed1e5938558345edaccaa79fd9ed26b9 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:44:45.221921 8884 raft_consensus.cc:686] T 6c0757c73551430b98c92b68aa0b2c05 P ed1e5938558345edaccaa79fd9ed26b9 [term 1 LEADER]: Becoming Leader. State: Replica: ed1e5938558345edaccaa79fd9ed26b9, State: Running, Role: LEADER
I20260926 03:44:45.222462 8884 consensus_queue.cc:237] T 6c0757c73551430b98c92b68aa0b2c05 P ed1e5938558345edaccaa79fd9ed26b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b25e3df61ce40b1818bdb428b3d47a1" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 45461 } } peers { permanent_uuid: "ae87bdf37fd34fd5b5dc9416b1878954" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 34427 } } peers { permanent_uuid: "ed1e5938558345edaccaa79fd9ed26b9" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 40371 } }
I20260926 03:44:45.227732 8367 catalog_manager.cc:5495] T 6c0757c73551430b98c92b68aa0b2c05 P ed1e5938558345edaccaa79fd9ed26b9 reported cstate change: term changed from 0 to 1, leader changed from <none> to ed1e5938558345edaccaa79fd9ed26b9 (127.0.194.193). New cstate: current_term: 1 leader_uuid: "ed1e5938558345edaccaa79fd9ed26b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b25e3df61ce40b1818bdb428b3d47a1" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 45461 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae87bdf37fd34fd5b5dc9416b1878954" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 34427 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ed1e5938558345edaccaa79fd9ed26b9" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 40371 } health_report { overall_health: HEALTHY } } }
I20260926 03:44:45.261788 8876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:45.256241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42850 (local address 127.0.194.193:40371)
0926 03:44:45.256426 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0926 03:44:45.256433 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:44:45.256452 (+ 19us) server_negotiation.cc:407] Connection header received
0926 03:44:45.256634 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:44:45.256639 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:44:45.256734 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:44:45.256887 (+ 153us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:44:45.257243 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:45.257927 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:45.259446 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:45.259843 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:45.259973 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:45.260384 (+ 411us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:44:45.261075 (+ 691us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:44:45.261280 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0926 03:44:45.261339 (+ 59us) server_negotiation.cc:299] Negotiation successful
0926 03:44:45.261448 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 03:44:45.269835 8664 raft_consensus.cc:1261] T 6c0757c73551430b98c92b68aa0b2c05 P ae87bdf37fd34fd5b5dc9416b1878954 [term 1 FOLLOWER]: Refusing update from remote peer ed1e5938558345edaccaa79fd9ed26b9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:44:45.269897 8807 raft_consensus.cc:1261] T 6c0757c73551430b98c92b68aa0b2c05 P 8b25e3df61ce40b1818bdb428b3d47a1 [term 1 FOLLOWER]: Refusing update from remote peer ed1e5938558345edaccaa79fd9ed26b9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:44:45.270821 8888 consensus_queue.cc:1035] T 6c0757c73551430b98c92b68aa0b2c05 P ed1e5938558345edaccaa79fd9ed26b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae87bdf37fd34fd5b5dc9416b1878954" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 34427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:44:45.271159 8884 consensus_queue.cc:1035] T 6c0757c73551430b98c92b68aa0b2c05 P ed1e5938558345edaccaa79fd9ed26b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b25e3df61ce40b1818bdb428b3d47a1" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 45461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:44:45.284559 8891 mvcc.cc:205] Tried to move back new op lower bound from 7333454992455950336 to 7333454992275238912. Current Snapshot: MvccSnapshot[applied={T|T < 7333454992455950336}]
I20260926 03:44:45.297315 8367 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56336:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:44:45.297575 8367 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56336:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:44:45.300973 8367 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f8f4c2ad1ed14a06a9cac074539f400f: Sending DeleteTablet for 3 replicas of tablet 6c0757c73551430b98c92b68aa0b2c05
I20260926 03:44:45.302121 8504 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c0757c73551430b98c92b68aa0b2c05 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:44:45 UTC) from {username='oryx', principal='oryx/127.0.194.254@KRBTEST.COM'} at 127.0.0.1:42844
I20260926 03:44:45.302104 8786 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c0757c73551430b98c92b68aa0b2c05 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:44:45 UTC) from {username='oryx', principal='oryx/127.0.194.254@KRBTEST.COM'} at 127.0.0.1:47720
I20260926 03:44:45.302168 8643 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c0757c73551430b98c92b68aa0b2c05 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:44:45 UTC) from {username='oryx', principal='oryx/127.0.194.254@KRBTEST.COM'} at 127.0.0.1:45630
I20260926 03:44:45.304804 8366 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56336:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 03:44:45.305872 8366 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 03:44:45.315938 8503 tablet_service.cc:1467] Processing CreateTablet for tablet 1e1e4259370f4de4a4f9ec16a291796f (DEFAULT_TABLE table=test-table [id=a79aecd0cd0a46b9b31299a6a897e34b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:44:45.316161 8642 tablet_service.cc:1467] Processing CreateTablet for tablet 1e1e4259370f4de4a4f9ec16a291796f (DEFAULT_TABLE table=test-table [id=a79aecd0cd0a46b9b31299a6a897e34b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:44:45.316362 8784 tablet_service.cc:1467] Processing CreateTablet for tablet 1e1e4259370f4de4a4f9ec16a291796f (DEFAULT_TABLE table=test-table [id=a79aecd0cd0a46b9b31299a6a897e34b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:44:45.316510 8503 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e1e4259370f4de4a4f9ec16a291796f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:44:45.316852 8784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e1e4259370f4de4a4f9ec16a291796f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:44:45.316846 8642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e1e4259370f4de4a4f9ec16a291796f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:44:45.323451 8878 tablet_bootstrap.cc:492] T 1e1e4259370f4de4a4f9ec16a291796f P ed1e5938558345edaccaa79fd9ed26b9: Bootstrap starting.
I20260926 03:44:45.324039 8877 tablet_bootstrap.cc:492] T 1e1e4259370f4de4a4f9ec16a291796f P 8b25e3df61ce40b1818bdb428b3d47a1: Bootstrap starting.
I20260926 03:44:45.325206 8878 tablet_bootstrap.cc:654] T 1e1e4259370f4de4a4f9ec16a291796f P ed1e5938558345edaccaa79fd9ed26b9: Neither blocks nor log segments found. Creating new log.
I20260926 03:44:45.325711 8877 tablet_bootstrap.cc:654] T 1e1e4259370f4de4a4f9ec16a291796f P 8b25e3df61ce40b1818bdb428b3d47a1: Neither blocks nor log segments found. Creating new log.
I20260926 03:44:45.328346 8902 tablet_replica.cc:330] T 6c0757c73551430b98c92b68aa0b2c05 P ed1e5938558345edaccaa79fd9ed26b9: stopping tablet replica
I20260926 03:44:45.329090 8902 raft_consensus.cc:2229] T 6c0757c73551430b98c92b68aa0b2c05 P ed1e5938558345edaccaa79fd9ed26b9 [term 1 LEADER]: Raft consensus shutting down.
I20260926 03:44:45.330665 8879 tablet_bootstrap.cc:492] T 1e1e4259370f4de4a4f9ec16a291796f P ae87bdf37fd34fd5b5dc9416b1878954: Bootstrap starting.
I20260926 03:44:45.331264 8902 raft_consensus.cc:2258] T 6c0757c73551430b98c92b68aa0b2c05 P ed1e5938558345edaccaa79fd9ed26b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:44:45.332139 8901 tablet_replica.cc:330] T 6c0757c73551430b98c92b68aa0b2c05 P 8b25e3df61ce40b1818bdb428b3d47a1: stopping tablet replica
I20260926 03:44:45.332370 8879 tablet_bootstrap.cc:654] T 1e1e4259370f4de4a4f9ec16a291796f P ae87bdf37fd34fd5b5dc9416b1878954: Neither blocks nor log segments found. Creating new log.
I20260926 03:44:45.333935 8901 raft_consensus.cc:2229] T 6c0757c73551430b98c92b68aa0b2c05 P 8b25e3df61ce40b1818bdb428b3d47a1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:44:45.334628 8901 raft_consensus.cc:2258] T 6c0757c73551430b98c92b68aa0b2c05 P 8b25e3df61ce40b1818bdb428b3d47a1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:44:45.337926 8901 ts_tablet_manager.cc:1905] T 6c0757c73551430b98c92b68aa0b2c05 P 8b25e3df61ce40b1818bdb428b3d47a1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:44:45.337926 8902 ts_tablet_manager.cc:1905] T 6c0757c73551430b98c92b68aa0b2c05 P ed1e5938558345edaccaa79fd9ed26b9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:44:45.338240 8877 tablet_bootstrap.cc:492] T 1e1e4259370f4de4a4f9ec16a291796f P 8b25e3df61ce40b1818bdb428b3d47a1: No bootstrap required, opened a new log
I20260926 03:44:45.338387 8877 ts_tablet_manager.cc:1397] T 1e1e4259370f4de4a4f9ec16a291796f P 8b25e3df61ce40b1818bdb428b3d47a1: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.007s
I20260926 03:44:45.339293 8877 raft_consensus.cc:348] T 1e1e4259370f4de4a4f9ec16a291796f P 8b25e3df61ce40b1818bdb428b3d47a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed1e5938558345edaccaa79fd9ed26b9" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 40371 } } peers { permanent_uuid: "ae87bdf37fd34fd5b5dc9416b1878954" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 34427 } } peers { permanent_uuid: "8b25e3df61ce40b1818bdb428b3d47a1" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 45461 } }
I20260926 03:44:45.339531 8877 raft_consensus.cc:374] T 1e1e4259370f4de4a4f9ec16a291796f P 8b25e3df61ce40b1818bdb428b3d47a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:44:45.339627 8877 raft_consensus.cc:729] T 1e1e4259370f4de4a4f9ec16a291796f P 8b25e3df61ce40b1818bdb428b3d47a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b25e3df61ce40b1818bdb428b3d47a1, State: Initialized, Role: FOLLOWER
I20260926 03:44:45.340055 8877 consensus_queue.cc:260] T 1e1e4259370f4de4a4f9ec16a291796f P 8b25e3df61ce40b1818bdb428b3d47a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed1e5938558345edaccaa79fd9ed26b9" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 40371 } } peers { permanent_uuid: "ae87bdf37fd34fd5b5dc9416b1878954" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 34427 } } peers { permanent_uuid: "8b25e3df61ce40b1818bdb428b3d47a1" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 45461 } }
I20260926 03:44:45.343025 8877 ts_tablet_manager.cc:1428] T 1e1e4259370f4de4a4f9ec16a291796f P 8b25e3df61ce40b1818bdb428b3d47a1: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260926 03:44:45.346421 8878 tablet_bootstrap.cc:492] T 1e1e4259370f4de4a4f9ec16a291796f P ed1e5938558345edaccaa79fd9ed26b9: No bootstrap required, opened a new log
I20260926 03:44:45.346588 8878 ts_tablet_manager.cc:1397] T 1e1e4259370f4de4a4f9ec16a291796f P ed1e5938558345edaccaa79fd9ed26b9: Time spent bootstrapping tablet: real 0.023s user 0.004s sys 0.006s
I20260926 03:44:45.346835 8903 tablet_replica.cc:330] T 6c0757c73551430b98c92b68aa0b2c05 P ae87bdf37fd34fd5b5dc9416b1878954: stopping tablet replica
I20260926 03:44:45.347345 8903 raft_consensus.cc:2229] T 6c0757c73551430b98c92b68aa0b2c05 P ae87bdf37fd34fd5b5dc9416b1878954 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:44:45.347486 8878 raft_consensus.cc:348] T 1e1e4259370f4de4a4f9ec16a291796f P ed1e5938558345edaccaa79fd9ed26b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed1e5938558345edaccaa79fd9ed26b9" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 40371 } } peers { permanent_uuid: "ae87bdf37fd34fd5b5dc9416b1878954" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 34427 } } peers { permanent_uuid: "8b25e3df61ce40b1818bdb428b3d47a1" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 45461 } }
I20260926 03:44:45.347715 8878 raft_consensus.cc:374] T 1e1e4259370f4de4a4f9ec16a291796f P ed1e5938558345edaccaa79fd9ed26b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:44:45.347813 8878 raft_consensus.cc:729] T 1e1e4259370f4de4a4f9ec16a291796f P ed1e5938558345edaccaa79fd9ed26b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed1e5938558345edaccaa79fd9ed26b9, State: Initialized, Role: FOLLOWER
I20260926 03:44:45.347844 8903 raft_consensus.cc:2258] T 6c0757c73551430b98c92b68aa0b2c05 P ae87bdf37fd34fd5b5dc9416b1878954 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:44:45.348096 8878 consensus_queue.cc:260] T 1e1e4259370f4de4a4f9ec16a291796f P ed1e5938558345edaccaa79fd9ed26b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed1e5938558345edaccaa79fd9ed26b9" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 40371 } } peers { permanent_uuid: "ae87bdf37fd34fd5b5dc9416b1878954" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 34427 } } peers { permanent_uuid: "8b25e3df61ce40b1818bdb428b3d47a1" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 45461 } }
I20260926 03:44:45.349586 8879 tablet_bootstrap.cc:492] T 1e1e4259370f4de4a4f9ec16a291796f P ae87bdf37fd34fd5b5dc9416b1878954: No bootstrap required, opened a new log
I20260926 03:44:45.349725 8879 ts_tablet_manager.cc:1397] T 1e1e4259370f4de4a4f9ec16a291796f P ae87bdf37fd34fd5b5dc9416b1878954: Time spent bootstrapping tablet: real 0.019s user 0.003s sys 0.007s
I20260926 03:44:45.349759 8901 ts_tablet_manager.cc:1918] T 6c0757c73551430b98c92b68aa0b2c05 P 8b25e3df61ce40b1818bdb428b3d47a1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:44:45.350039 8901 log.cc:1196] T 6c0757c73551430b98c92b68aa0b2c05 P 8b25e3df61ce40b1818bdb428b3d47a1: Deleting WAL directory at /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790394221564982-779-0/minicluster-data/ts-2/wal/wals/6c0757c73551430b98c92b68aa0b2c05
I20260926 03:44:45.350666 8901 ts_tablet_manager.cc:1939] T 6c0757c73551430b98c92b68aa0b2c05 P 8b25e3df61ce40b1818bdb428b3d47a1: Deleting consensus metadata
I20260926 03:44:45.350585 8879 raft_consensus.cc:348] T 1e1e4259370f4de4a4f9ec16a291796f P ae87bdf37fd34fd5b5dc9416b1878954 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed1e5938558345edaccaa79fd9ed26b9" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 40371 } } peers { permanent_uuid: "ae87bdf37fd34fd5b5dc9416b1878954" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 34427 } } peers { permanent_uuid: "8b25e3df61ce40b1818bdb428b3d47a1" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 45461 } }
I20260926 03:44:45.350836 8879 raft_consensus.cc:374] T 1e1e4259370f4de4a4f9ec16a291796f P ae87bdf37fd34fd5b5dc9416b1878954 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:44:45.350852 8878 ts_tablet_manager.cc:1428] T 1e1e4259370f4de4a4f9ec16a291796f P ed1e5938558345edaccaa79fd9ed26b9: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260926 03:44:45.350929 8879 raft_consensus.cc:729] T 1e1e4259370f4de4a4f9ec16a291796f P ae87bdf37fd34fd5b5dc9416b1878954 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae87bdf37fd34fd5b5dc9416b1878954, State: Initialized, Role: FOLLOWER
I20260926 03:44:45.351219 8879 consensus_queue.cc:260] T 1e1e4259370f4de4a4f9ec16a291796f P ae87bdf37fd34fd5b5dc9416b1878954 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed1e5938558345edaccaa79fd9ed26b9" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 40371 } } peers { permanent_uuid: "ae87bdf37fd34fd5b5dc9416b1878954" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 34427 } } peers { permanent_uuid: "8b25e3df61ce40b1818bdb428b3d47a1" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 45461 } }
I20260926 03:44:45.351756 8879 ts_tablet_manager.cc:1428] T 1e1e4259370f4de4a4f9ec16a291796f P ae87bdf37fd34fd5b5dc9416b1878954: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 03:44:45.354506 8902 ts_tablet_manager.cc:1918] T 6c0757c73551430b98c92b68aa0b2c05 P ed1e5938558345edaccaa79fd9ed26b9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:44:45.354669 8902 log.cc:1196] T 6c0757c73551430b98c92b68aa0b2c05 P ed1e5938558345edaccaa79fd9ed26b9: Deleting WAL directory at /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790394221564982-779-0/minicluster-data/ts-0/wal/wals/6c0757c73551430b98c92b68aa0b2c05
I20260926 03:44:45.354820 8351 catalog_manager.cc:4841] TS 8b25e3df61ce40b1818bdb428b3d47a1 (127.0.194.195:45461): tablet 6c0757c73551430b98c92b68aa0b2c05 (table test-table [id=15f0d2c0750b4aa483d63d93c3b997cc]) successfully deleted
I20260926 03:44:45.355265 8902 ts_tablet_manager.cc:1939] T 6c0757c73551430b98c92b68aa0b2c05 P ed1e5938558345edaccaa79fd9ed26b9: Deleting consensus metadata
I20260926 03:44:45.356920 8354 catalog_manager.cc:4841] TS ed1e5938558345edaccaa79fd9ed26b9 (127.0.194.193:40371): tablet 6c0757c73551430b98c92b68aa0b2c05 (table test-table [id=15f0d2c0750b4aa483d63d93c3b997cc]) successfully deleted
I20260926 03:44:45.357883 8903 ts_tablet_manager.cc:1905] T 6c0757c73551430b98c92b68aa0b2c05 P ae87bdf37fd34fd5b5dc9416b1878954: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:44:45.364624 8903 ts_tablet_manager.cc:1918] T 6c0757c73551430b98c92b68aa0b2c05 P ae87bdf37fd34fd5b5dc9416b1878954: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:44:45.364775 8903 log.cc:1196] T 6c0757c73551430b98c92b68aa0b2c05 P ae87bdf37fd34fd5b5dc9416b1878954: Deleting WAL directory at /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790394221564982-779-0/minicluster-data/ts-1/wal/wals/6c0757c73551430b98c92b68aa0b2c05
I20260926 03:44:45.365269 8903 ts_tablet_manager.cc:1939] T 6c0757c73551430b98c92b68aa0b2c05 P ae87bdf37fd34fd5b5dc9416b1878954: Deleting consensus metadata
I20260926 03:44:45.366896 8353 catalog_manager.cc:4841] TS ae87bdf37fd34fd5b5dc9416b1878954 (127.0.194.194:34427): tablet 6c0757c73551430b98c92b68aa0b2c05 (table test-table [id=15f0d2c0750b4aa483d63d93c3b997cc]) successfully deleted
W20260926 03:44:45.376080 8575 tablet.cc:2369] T 1e1e4259370f4de4a4f9ec16a291796f P ed1e5938558345edaccaa79fd9ed26b9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 03:44:45.398367 8714 tablet.cc:2369] T 1e1e4259370f4de4a4f9ec16a291796f P ae87bdf37fd34fd5b5dc9416b1878954: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 03:44:45.431102 8853 tablet.cc:2369] T 1e1e4259370f4de4a4f9ec16a291796f P 8b25e3df61ce40b1818bdb428b3d47a1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 03:44:45.470564 8883 raft_consensus.cc:482] T 1e1e4259370f4de4a4f9ec16a291796f P 8b25e3df61ce40b1818bdb428b3d47a1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 03:44:45.470873 8883 raft_consensus.cc:504] T 1e1e4259370f4de4a4f9ec16a291796f P 8b25e3df61ce40b1818bdb428b3d47a1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed1e5938558345edaccaa79fd9ed26b9" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 40371 } } peers { permanent_uuid: "ae87bdf37fd34fd5b5dc9416b1878954" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 34427 } } peers { permanent_uuid: "8b25e3df61ce40b1818bdb428b3d47a1" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 45461 } }
I20260926 03:44:45.472779 8883 leader_election.cc:288] T 1e1e4259370f4de4a4f9ec16a291796f P 8b25e3df61ce40b1818bdb428b3d47a1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ed1e5938558345edaccaa79fd9ed26b9 (127.0.194.193:40371), ae87bdf37fd34fd5b5dc9416b1878954 (127.0.194.194:34427)
I20260926 03:44:45.481961 8874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:45.473349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.195:48583 (local address 127.0.194.194:34427)
0926 03:44:45.473543 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0926 03:44:45.473550 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:44:45.474968 (+ 1418us) server_negotiation.cc:407] Connection header received
0926 03:44:45.475255 (+ 287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:44:45.475261 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:44:45.475342 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:44:45.475415 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:44:45.476437 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:45.477536 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:45.479789 (+ 2253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:45.480781 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:45.480944 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:45.481014 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 03:44:45.481537 (+ 523us) server_negotiation.cc:299] Negotiation successful
0926 03:44:45.481678 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 03:44:45.482072 8876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:45.473680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.195:57559 (local address 127.0.194.193:40371)
0926 03:44:45.473844 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0926 03:44:45.473850 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:44:45.474493 (+ 643us) server_negotiation.cc:407] Connection header received
0926 03:44:45.474875 (+ 382us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:44:45.474881 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:44:45.474988 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:44:45.475119 (+ 131us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:44:45.476194 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:45.477032 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:45.479494 (+ 2462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:45.480861 (+ 1367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:45.481029 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:45.481103 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 03:44:45.481630 (+ 527us) server_negotiation.cc:299] Negotiation successful
0926 03:44:45.481766 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 03:44:45.482638 8907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:45.473150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.194:34427 (local address 127.0.194.195:48583)
0926 03:44:45.474857 (+ 1707us) negotiation.cc:107] Waiting for socket to connect
0926 03:44:45.474876 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 03:44:45.475077 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:44:45.475664 (+ 587us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:44:45.475671 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:44:45.475709 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:44:45.476248 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:45.476264 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:45.477675 (+ 1411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:45.477680 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:45.479654 (+ 1974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:45.479663 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:45.481170 (+ 1507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:45.481179 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:45.481355 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:45.481387 (+ 32us) client_negotiation.cc:769] Sending connection context
0926 03:44:45.482204 (+ 817us) client_negotiation.cc:240] Negotiation successful
0926 03:44:45.482341 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1553,"mutex_wait_us":28,"thread_start_us":901,"threads_started":1}
I20260926 03:44:45.483145 8908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:45.473020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.193:40371 (local address 127.0.194.195:57559)
0926 03:44:45.474371 (+ 1351us) negotiation.cc:107] Waiting for socket to connect
0926 03:44:45.474391 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 03:44:45.474607 (+ 216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:44:45.475338 (+ 731us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:44:45.475343 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:44:45.475395 (+ 52us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:44:45.476004 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:45.476018 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:45.477176 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:45.477182 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:45.479324 (+ 2142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:45.479343 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:45.481254 (+ 1911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:45.481260 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:45.481441 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:45.481473 (+ 32us) client_negotiation.cc:769] Sending connection context
0926 03:44:45.482590 (+ 1117us) client_negotiation.cc:240] Negotiation successful
0926 03:44:45.482765 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1216,"mutex_wait_us":7,"thread_start_us":1438,"threads_started":1}
I20260926 03:44:45.483371 8664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e1e4259370f4de4a4f9ec16a291796f" candidate_uuid: "8b25e3df61ce40b1818bdb428b3d47a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae87bdf37fd34fd5b5dc9416b1878954" is_pre_election: true
I20260926 03:44:45.483629 8664 raft_consensus.cc:2454] T 1e1e4259370f4de4a4f9ec16a291796f P ae87bdf37fd34fd5b5dc9416b1878954 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b25e3df61ce40b1818bdb428b3d47a1 in term 0.
I20260926 03:44:45.484133 8735 leader_election.cc:302] T 1e1e4259370f4de4a4f9ec16a291796f P 8b25e3df61ce40b1818bdb428b3d47a1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b25e3df61ce40b1818bdb428b3d47a1, ae87bdf37fd34fd5b5dc9416b1878954; no voters:
I20260926 03:44:45.484025 8524 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e1e4259370f4de4a4f9ec16a291796f" candidate_uuid: "8b25e3df61ce40b1818bdb428b3d47a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed1e5938558345edaccaa79fd9ed26b9" is_pre_election: true
I20260926 03:44:45.484505 8524 raft_consensus.cc:2454] T 1e1e4259370f4de4a4f9ec16a291796f P ed1e5938558345edaccaa79fd9ed26b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b25e3df61ce40b1818bdb428b3d47a1 in term 0.
I20260926 03:44:45.484953 8883 raft_consensus.cc:2781] T 1e1e4259370f4de4a4f9ec16a291796f P 8b25e3df61ce40b1818bdb428b3d47a1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 03:44:45.485083 8883 raft_consensus.cc:482] T 1e1e4259370f4de4a4f9ec16a291796f P 8b25e3df61ce40b1818bdb428b3d47a1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 03:44:45.485199 8883 raft_consensus.cc:3037] T 1e1e4259370f4de4a4f9ec16a291796f P 8b25e3df61ce40b1818bdb428b3d47a1 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:44:45.488457 8883 raft_consensus.cc:504] T 1e1e4259370f4de4a4f9ec16a291796f P 8b25e3df61ce40b1818bdb428b3d47a1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed1e5938558345edaccaa79fd9ed26b9" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 40371 } } peers { permanent_uuid: "ae87bdf37fd34fd5b5dc9416b1878954" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 34427 } } peers { permanent_uuid: "8b25e3df61ce40b1818bdb428b3d47a1" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 45461 } }
I20260926 03:44:45.489188 8883 leader_election.cc:288] T 1e1e4259370f4de4a4f9ec16a291796f P 8b25e3df61ce40b1818bdb428b3d47a1 [CANDIDATE]: Term 1 election: Requested vote from peers ed1e5938558345edaccaa79fd9ed26b9 (127.0.194.193:40371), ae87bdf37fd34fd5b5dc9416b1878954 (127.0.194.194:34427)
I20260926 03:44:45.489672 8524 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e1e4259370f4de4a4f9ec16a291796f" candidate_uuid: "8b25e3df61ce40b1818bdb428b3d47a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed1e5938558345edaccaa79fd9ed26b9"
I20260926 03:44:45.489704 8664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e1e4259370f4de4a4f9ec16a291796f" candidate_uuid: "8b25e3df61ce40b1818bdb428b3d47a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae87bdf37fd34fd5b5dc9416b1878954"
I20260926 03:44:45.489843 8524 raft_consensus.cc:3037] T 1e1e4259370f4de4a4f9ec16a291796f P ed1e5938558345edaccaa79fd9ed26b9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:44:45.489903 8664 raft_consensus.cc:3037] T 1e1e4259370f4de4a4f9ec16a291796f P ae87bdf37fd34fd5b5dc9416b1878954 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:44:45.492179 8524 raft_consensus.cc:2454] T 1e1e4259370f4de4a4f9ec16a291796f P ed1e5938558345edaccaa79fd9ed26b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b25e3df61ce40b1818bdb428b3d47a1 in term 1.
I20260926 03:44:45.492662 8736 leader_election.cc:302] T 1e1e4259370f4de4a4f9ec16a291796f P 8b25e3df61ce40b1818bdb428b3d47a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b25e3df61ce40b1818bdb428b3d47a1, ed1e5938558345edaccaa79fd9ed26b9; no voters:
I20260926 03:44:45.492888 8664 raft_consensus.cc:2454] T 1e1e4259370f4de4a4f9ec16a291796f P ae87bdf37fd34fd5b5dc9416b1878954 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b25e3df61ce40b1818bdb428b3d47a1 in term 1.
I20260926 03:44:45.493001 8883 raft_consensus.cc:2781] T 1e1e4259370f4de4a4f9ec16a291796f P 8b25e3df61ce40b1818bdb428b3d47a1 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:44:45.493743 8883 raft_consensus.cc:686] T 1e1e4259370f4de4a4f9ec16a291796f P 8b25e3df61ce40b1818bdb428b3d47a1 [term 1 LEADER]: Becoming Leader. State: Replica: 8b25e3df61ce40b1818bdb428b3d47a1, State: Running, Role: LEADER
I20260926 03:44:45.494158 8883 consensus_queue.cc:237] T 1e1e4259370f4de4a4f9ec16a291796f P 8b25e3df61ce40b1818bdb428b3d47a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed1e5938558345edaccaa79fd9ed26b9" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 40371 } } peers { permanent_uuid: "ae87bdf37fd34fd5b5dc9416b1878954" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 34427 } } peers { permanent_uuid: "8b25e3df61ce40b1818bdb428b3d47a1" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 45461 } }
I20260926 03:44:45.497517 8366 catalog_manager.cc:5495] T 1e1e4259370f4de4a4f9ec16a291796f P 8b25e3df61ce40b1818bdb428b3d47a1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8b25e3df61ce40b1818bdb428b3d47a1 (127.0.194.195). New cstate: current_term: 1 leader_uuid: "8b25e3df61ce40b1818bdb428b3d47a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed1e5938558345edaccaa79fd9ed26b9" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 40371 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae87bdf37fd34fd5b5dc9416b1878954" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 34427 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b25e3df61ce40b1818bdb428b3d47a1" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 45461 } health_report { overall_health: HEALTHY } } }
Sep 26 03:44:45 dist-test-slave-n2th krb5kdc[8322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394282, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.194.254@KRBTEST.COM for oryx/127.0.194.254@KRBTEST.COM
I20260926 03:44:45.560905 8916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:45.545283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:34071 (local address 127.0.0.1:56342)
0926 03:44:45.545959 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0926 03:44:45.545980 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 03:44:45.546099 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:44:45.546495 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:44:45.546498 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:44:45.547201 (+ 703us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:44:45.547504 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:45.547513 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:45.548979 (+ 1466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:45.548982 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:45.550144 (+ 1162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:45.550155 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:45.551131 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:45.551136 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:45.551248 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:45.551945 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:44:45.551963 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:44:45.554239 (+ 2276us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:44:45.558129 (+ 3890us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:44:45.558141 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:44:45.558148 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:44:45.558852 (+ 704us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:44:45.559252 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:44:45.559256 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:44:45.559258 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:44:45.559408 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:44:45.560091 (+ 683us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:44:45.560097 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:44:45.560266 (+ 169us) client_negotiation.cc:769] Sending connection context
0926 03:44:45.560402 (+ 136us) client_negotiation.cc:240] Negotiation successful
0926 03:44:45.560605 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":548,"thread_start_us":362,"threads_started":1}
I20260926 03:44:45.561180 8462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:45.545257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56342 (local address 127.0.194.254:34071)
0926 03:44:45.545483 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0926 03:44:45.545491 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:44:45.546052 (+ 561us) server_negotiation.cc:407] Connection header received
0926 03:44:45.546226 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:44:45.546233 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:44:45.546311 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:44:45.546447 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:44:45.547649 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:45.548855 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:45.550277 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:45.550958 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:45.551092 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:45.554436 (+ 3344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:44:45.554443 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:44:45.554448 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:44:45.554495 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:44:45.557926 (+ 3431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:45.558975 (+ 1049us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:45.558982 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:45.558987 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:45.559131 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:45.559529 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:45.559534 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:45.559538 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:45.559927 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:44:45.560108 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0926 03:44:45.560541 (+ 433us) server_negotiation.cc:299] Negotiation successful
0926 03:44:45.560771 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260926 03:44:45.566488 8366 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.194.254@KRBTEST.COM'} at 127.0.0.1:56342:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260926 03:44:45.567679 8366 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 03:44:45.575498 8504 tablet_service.cc:1467] Processing CreateTablet for tablet b57728c01421438dac89d15fea521318 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1ecca12f31134fbf86c64f1902b64df2]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 03:44:45.575836 8786 tablet_service.cc:1467] Processing CreateTablet for tablet b57728c01421438dac89d15fea521318 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1ecca12f31134fbf86c64f1902b64df2]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 03:44:45.575886 8642 tablet_service.cc:1467] Processing CreateTablet for tablet b57728c01421438dac89d15fea521318 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1ecca12f31134fbf86c64f1902b64df2]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 03:44:45.576160 8504 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b57728c01421438dac89d15fea521318. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:44:45.576341 8786 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b57728c01421438dac89d15fea521318. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:44:45.576370 8642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b57728c01421438dac89d15fea521318. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:44:45.583858 8877 tablet_bootstrap.cc:492] T b57728c01421438dac89d15fea521318 P 8b25e3df61ce40b1818bdb428b3d47a1: Bootstrap starting.
I20260926 03:44:45.583873 8878 tablet_bootstrap.cc:492] T b57728c01421438dac89d15fea521318 P ed1e5938558345edaccaa79fd9ed26b9: Bootstrap starting.
I20260926 03:44:45.585218 8877 tablet_bootstrap.cc:654] T b57728c01421438dac89d15fea521318 P 8b25e3df61ce40b1818bdb428b3d47a1: Neither blocks nor log segments found. Creating new log.
I20260926 03:44:45.585736 8878 tablet_bootstrap.cc:654] T b57728c01421438dac89d15fea521318 P ed1e5938558345edaccaa79fd9ed26b9: Neither blocks nor log segments found. Creating new log.
I20260926 03:44:45.587493 8877 tablet_bootstrap.cc:492] T b57728c01421438dac89d15fea521318 P 8b25e3df61ce40b1818bdb428b3d47a1: No bootstrap required, opened a new log
I20260926 03:44:45.587669 8877 ts_tablet_manager.cc:1397] T b57728c01421438dac89d15fea521318 P 8b25e3df61ce40b1818bdb428b3d47a1: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260926 03:44:45.588234 8877 raft_consensus.cc:348] T b57728c01421438dac89d15fea521318 P 8b25e3df61ce40b1818bdb428b3d47a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed1e5938558345edaccaa79fd9ed26b9" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 40371 } } peers { permanent_uuid: "ae87bdf37fd34fd5b5dc9416b1878954" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 34427 } } peers { permanent_uuid: "8b25e3df61ce40b1818bdb428b3d47a1" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 45461 } }
I20260926 03:44:45.588462 8877 raft_consensus.cc:374] T b57728c01421438dac89d15fea521318 P 8b25e3df61ce40b1818bdb428b3d47a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:44:45.588560 8877 raft_consensus.cc:729] T b57728c01421438dac89d15fea521318 P 8b25e3df61ce40b1818bdb428b3d47a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b25e3df61ce40b1818bdb428b3d47a1, State: Initialized, Role: FOLLOWER
I20260926 03:44:45.588797 8877 consensus_queue.cc:260] T b57728c01421438dac89d15fea521318 P 8b25e3df61ce40b1818bdb428b3d47a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed1e5938558345edaccaa79fd9ed26b9" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 40371 } } peers { permanent_uuid: "ae87bdf37fd34fd5b5dc9416b1878954" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 34427 } } peers { permanent_uuid: "8b25e3df61ce40b1818bdb428b3d47a1" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 45461 } }
I20260926 03:44:45.589167 8883 tablet_replica.cc:439] T b57728c01421438dac89d15fea521318 P 8b25e3df61ce40b1818bdb428b3d47a1: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed1e5938558345edaccaa79fd9ed26b9" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 40371 } } peers { permanent_uuid: "ae87bdf37fd34fd5b5dc9416b1878954" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 34427 } } peers { permanent_uuid: "8b25e3df61ce40b1818bdb428b3d47a1" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 45461 } } }
I20260926 03:44:45.589429 8883 tablet_replica.cc:442] T b57728c01421438dac89d15fea521318 P 8b25e3df61ce40b1818bdb428b3d47a1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 03:44:45.590121 8879 tablet_bootstrap.cc:492] T b57728c01421438dac89d15fea521318 P ae87bdf37fd34fd5b5dc9416b1878954: Bootstrap starting.
I20260926 03:44:45.590183 8877 ts_tablet_manager.cc:1428] T b57728c01421438dac89d15fea521318 P 8b25e3df61ce40b1818bdb428b3d47a1: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 03:44:45.591925 8879 tablet_bootstrap.cc:654] T b57728c01421438dac89d15fea521318 P ae87bdf37fd34fd5b5dc9416b1878954: Neither blocks nor log segments found. Creating new log.
I20260926 03:44:45.596969 8878 tablet_bootstrap.cc:492] T b57728c01421438dac89d15fea521318 P ed1e5938558345edaccaa79fd9ed26b9: No bootstrap required, opened a new log
I20260926 03:44:45.597190 8878 ts_tablet_manager.cc:1397] T b57728c01421438dac89d15fea521318 P ed1e5938558345edaccaa79fd9ed26b9: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.004s
I20260926 03:44:45.597828 8878 raft_consensus.cc:348] T b57728c01421438dac89d15fea521318 P ed1e5938558345edaccaa79fd9ed26b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed1e5938558345edaccaa79fd9ed26b9" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 40371 } } peers { permanent_uuid: "ae87bdf37fd34fd5b5dc9416b1878954" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 34427 } } peers { permanent_uuid: "8b25e3df61ce40b1818bdb428b3d47a1" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 45461 } }
I20260926 03:44:45.598075 8878 raft_consensus.cc:374] T b57728c01421438dac89d15fea521318 P ed1e5938558345edaccaa79fd9ed26b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:44:45.598174 8878 raft_consensus.cc:729] T b57728c01421438dac89d15fea521318 P ed1e5938558345edaccaa79fd9ed26b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed1e5938558345edaccaa79fd9ed26b9, State: Initialized, Role: FOLLOWER
I20260926 03:44:45.598489 8878 consensus_queue.cc:260] T b57728c01421438dac89d15fea521318 P ed1e5938558345edaccaa79fd9ed26b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed1e5938558345edaccaa79fd9ed26b9" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 40371 } } peers { permanent_uuid: "ae87bdf37fd34fd5b5dc9416b1878954" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 34427 } } peers { permanent_uuid: "8b25e3df61ce40b1818bdb428b3d47a1" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 45461 } }
I20260926 03:44:45.598966 8888 tablet_replica.cc:439] T b57728c01421438dac89d15fea521318 P ed1e5938558345edaccaa79fd9ed26b9: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed1e5938558345edaccaa79fd9ed26b9" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 40371 } } peers { permanent_uuid: "ae87bdf37fd34fd5b5dc9416b1878954" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 34427 } } peers { permanent_uuid: "8b25e3df61ce40b1818bdb428b3d47a1" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 45461 } } }
I20260926 03:44:45.599340 8888 tablet_replica.cc:442] T b57728c01421438dac89d15fea521318 P ed1e5938558345edaccaa79fd9ed26b9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 03:44:45.600471 8878 ts_tablet_manager.cc:1428] T b57728c01421438dac89d15fea521318 P ed1e5938558345edaccaa79fd9ed26b9: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260926 03:44:45.606882 8879 tablet_bootstrap.cc:492] T b57728c01421438dac89d15fea521318 P ae87bdf37fd34fd5b5dc9416b1878954: No bootstrap required, opened a new log
I20260926 03:44:45.607075 8879 ts_tablet_manager.cc:1397] T b57728c01421438dac89d15fea521318 P ae87bdf37fd34fd5b5dc9416b1878954: Time spent bootstrapping tablet: real 0.017s user 0.012s sys 0.003s
I20260926 03:44:45.607825 8879 raft_consensus.cc:348] T b57728c01421438dac89d15fea521318 P ae87bdf37fd34fd5b5dc9416b1878954 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed1e5938558345edaccaa79fd9ed26b9" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 40371 } } peers { permanent_uuid: "ae87bdf37fd34fd5b5dc9416b1878954" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 34427 } } peers { permanent_uuid: "8b25e3df61ce40b1818bdb428b3d47a1" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 45461 } }
I20260926 03:44:45.608073 8879 raft_consensus.cc:374] T b57728c01421438dac89d15fea521318 P ae87bdf37fd34fd5b5dc9416b1878954 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:44:45.608158 8879 raft_consensus.cc:729] T b57728c01421438dac89d15fea521318 P ae87bdf37fd34fd5b5dc9416b1878954 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae87bdf37fd34fd5b5dc9416b1878954, State: Initialized, Role: FOLLOWER
I20260926 03:44:45.608402 8879 consensus_queue.cc:260] T b57728c01421438dac89d15fea521318 P ae87bdf37fd34fd5b5dc9416b1878954 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed1e5938558345edaccaa79fd9ed26b9" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 40371 } } peers { permanent_uuid: "ae87bdf37fd34fd5b5dc9416b1878954" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 34427 } } peers { permanent_uuid: "8b25e3df61ce40b1818bdb428b3d47a1" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 45461 } }
I20260926 03:44:45.609025 8879 ts_tablet_manager.cc:1428] T b57728c01421438dac89d15fea521318 P ae87bdf37fd34fd5b5dc9416b1878954: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260926 03:44:45.608908 8885 tablet_replica.cc:439] T b57728c01421438dac89d15fea521318 P ae87bdf37fd34fd5b5dc9416b1878954: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed1e5938558345edaccaa79fd9ed26b9" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 40371 } } peers { permanent_uuid: "ae87bdf37fd34fd5b5dc9416b1878954" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 34427 } } peers { permanent_uuid: "8b25e3df61ce40b1818bdb428b3d47a1" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 45461 } } }
I20260926 03:44:45.609287 8885 tablet_replica.cc:442] T b57728c01421438dac89d15fea521318 P ae87bdf37fd34fd5b5dc9416b1878954: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 03:44:45.723634 8888 raft_consensus.cc:482] T b57728c01421438dac89d15fea521318 P ed1e5938558345edaccaa79fd9ed26b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 03:44:45.723879 8888 raft_consensus.cc:504] T b57728c01421438dac89d15fea521318 P ed1e5938558345edaccaa79fd9ed26b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed1e5938558345edaccaa79fd9ed26b9" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 40371 } } peers { permanent_uuid: "ae87bdf37fd34fd5b5dc9416b1878954" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 34427 } } peers { permanent_uuid: "8b25e3df61ce40b1818bdb428b3d47a1" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 45461 } }
I20260926 03:44:45.724692 8888 leader_election.cc:288] T b57728c01421438dac89d15fea521318 P ed1e5938558345edaccaa79fd9ed26b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ae87bdf37fd34fd5b5dc9416b1878954 (127.0.194.194:34427), 8b25e3df61ce40b1818bdb428b3d47a1 (127.0.194.195:45461)
I20260926 03:44:45.725379 8664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b57728c01421438dac89d15fea521318" candidate_uuid: "ed1e5938558345edaccaa79fd9ed26b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae87bdf37fd34fd5b5dc9416b1878954" is_pre_election: true
I20260926 03:44:45.725399 8807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b57728c01421438dac89d15fea521318" candidate_uuid: "ed1e5938558345edaccaa79fd9ed26b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b25e3df61ce40b1818bdb428b3d47a1" is_pre_election: true
I20260926 03:44:45.725643 8807 raft_consensus.cc:2454] T b57728c01421438dac89d15fea521318 P 8b25e3df61ce40b1818bdb428b3d47a1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed1e5938558345edaccaa79fd9ed26b9 in term 0.
I20260926 03:44:45.725670 8664 raft_consensus.cc:2454] T b57728c01421438dac89d15fea521318 P ae87bdf37fd34fd5b5dc9416b1878954 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed1e5938558345edaccaa79fd9ed26b9 in term 0.
I20260926 03:44:45.726063 8456 leader_election.cc:302] T b57728c01421438dac89d15fea521318 P ed1e5938558345edaccaa79fd9ed26b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae87bdf37fd34fd5b5dc9416b1878954, ed1e5938558345edaccaa79fd9ed26b9; no voters:
I20260926 03:44:45.726377 8888 raft_consensus.cc:2781] T b57728c01421438dac89d15fea521318 P ed1e5938558345edaccaa79fd9ed26b9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 03:44:45.726496 8888 raft_consensus.cc:482] T b57728c01421438dac89d15fea521318 P ed1e5938558345edaccaa79fd9ed26b9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 03:44:45.726629 8888 raft_consensus.cc:3037] T b57728c01421438dac89d15fea521318 P ed1e5938558345edaccaa79fd9ed26b9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:44:45.730190 8888 raft_consensus.cc:504] T b57728c01421438dac89d15fea521318 P ed1e5938558345edaccaa79fd9ed26b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed1e5938558345edaccaa79fd9ed26b9" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 40371 } } peers { permanent_uuid: "ae87bdf37fd34fd5b5dc9416b1878954" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 34427 } } peers { permanent_uuid: "8b25e3df61ce40b1818bdb428b3d47a1" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 45461 } }
I20260926 03:44:45.731055 8888 leader_election.cc:288] T b57728c01421438dac89d15fea521318 P ed1e5938558345edaccaa79fd9ed26b9 [CANDIDATE]: Term 1 election: Requested vote from peers ae87bdf37fd34fd5b5dc9416b1878954 (127.0.194.194:34427), 8b25e3df61ce40b1818bdb428b3d47a1 (127.0.194.195:45461)
I20260926 03:44:45.731640 8664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b57728c01421438dac89d15fea521318" candidate_uuid: "ed1e5938558345edaccaa79fd9ed26b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae87bdf37fd34fd5b5dc9416b1878954"
I20260926 03:44:45.731680 8807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b57728c01421438dac89d15fea521318" candidate_uuid: "ed1e5938558345edaccaa79fd9ed26b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b25e3df61ce40b1818bdb428b3d47a1"
I20260926 03:44:45.731896 8664 raft_consensus.cc:3037] T b57728c01421438dac89d15fea521318 P ae87bdf37fd34fd5b5dc9416b1878954 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:44:45.731930 8807 raft_consensus.cc:3037] T b57728c01421438dac89d15fea521318 P 8b25e3df61ce40b1818bdb428b3d47a1 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:44:45.733911 8664 raft_consensus.cc:2454] T b57728c01421438dac89d15fea521318 P ae87bdf37fd34fd5b5dc9416b1878954 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed1e5938558345edaccaa79fd9ed26b9 in term 1.
I20260926 03:44:45.734444 8456 leader_election.cc:302] T b57728c01421438dac89d15fea521318 P ed1e5938558345edaccaa79fd9ed26b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae87bdf37fd34fd5b5dc9416b1878954, ed1e5938558345edaccaa79fd9ed26b9; no voters:
I20260926 03:44:45.734759 8888 raft_consensus.cc:2781] T b57728c01421438dac89d15fea521318 P ed1e5938558345edaccaa79fd9ed26b9 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:44:45.734889 8807 raft_consensus.cc:2454] T b57728c01421438dac89d15fea521318 P 8b25e3df61ce40b1818bdb428b3d47a1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed1e5938558345edaccaa79fd9ed26b9 in term 1.
I20260926 03:44:45.734967 8888 raft_consensus.cc:686] T b57728c01421438dac89d15fea521318 P ed1e5938558345edaccaa79fd9ed26b9 [term 1 LEADER]: Becoming Leader. State: Replica: ed1e5938558345edaccaa79fd9ed26b9, State: Running, Role: LEADER
I20260926 03:44:45.735229 8888 consensus_queue.cc:237] T b57728c01421438dac89d15fea521318 P ed1e5938558345edaccaa79fd9ed26b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed1e5938558345edaccaa79fd9ed26b9" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 40371 } } peers { permanent_uuid: "ae87bdf37fd34fd5b5dc9416b1878954" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 34427 } } peers { permanent_uuid: "8b25e3df61ce40b1818bdb428b3d47a1" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 45461 } }
I20260926 03:44:45.736937 8884 tablet_replica.cc:439] T b57728c01421438dac89d15fea521318 P ed1e5938558345edaccaa79fd9ed26b9: TxnStatusTablet state changed. Reason: New leader ed1e5938558345edaccaa79fd9ed26b9. Latest consensus state: current_term: 1 leader_uuid: "ed1e5938558345edaccaa79fd9ed26b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed1e5938558345edaccaa79fd9ed26b9" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 40371 } } peers { permanent_uuid: "ae87bdf37fd34fd5b5dc9416b1878954" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 34427 } } peers { permanent_uuid: "8b25e3df61ce40b1818bdb428b3d47a1" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 45461 } } }
I20260926 03:44:45.737205 8884 tablet_replica.cc:442] T b57728c01421438dac89d15fea521318 P ed1e5938558345edaccaa79fd9ed26b9: This TxnStatusTablet replica's current role is: LEADER
I20260926 03:44:45.738927 8366 catalog_manager.cc:5495] T b57728c01421438dac89d15fea521318 P ed1e5938558345edaccaa79fd9ed26b9 reported cstate change: term changed from 0 to 1, leader changed from <none> to ed1e5938558345edaccaa79fd9ed26b9 (127.0.194.193). New cstate: current_term: 1 leader_uuid: "ed1e5938558345edaccaa79fd9ed26b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed1e5938558345edaccaa79fd9ed26b9" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 40371 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae87bdf37fd34fd5b5dc9416b1878954" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 34427 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b25e3df61ce40b1818bdb428b3d47a1" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 45461 } health_report { overall_health: UNKNOWN } } }
I20260926 03:44:45.741955 8921 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260926 03:44:45.742221 8921 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260926 03:44:45.767935 8916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:45.762179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.193:40371 (local address 127.0.0.1:42862)
0926 03:44:45.762407 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0926 03:44:45.762428 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 03:44:45.762571 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:44:45.762973 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:44:45.762977 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:44:45.762987 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 03:44:45.763136 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:45.763143 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:45.764630 (+ 1487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:45.764635 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:45.765698 (+ 1063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:45.765710 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:45.766851 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:45.766856 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:45.766988 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:45.767044 (+ 56us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 03:44:45.767427 (+ 383us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 03:44:45.767442 (+ 15us) client_negotiation.cc:769] Sending connection context
0926 03:44:45.767542 (+ 100us) client_negotiation.cc:240] Negotiation successful
0926 03:44:45.767713 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260926 03:44:45.767951 8876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:45.762390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42862 (local address 127.0.194.193:40371)
0926 03:44:45.762611 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0926 03:44:45.762617 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:44:45.762640 (+ 23us) server_negotiation.cc:407] Connection header received
0926 03:44:45.762711 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:44:45.762716 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:44:45.762828 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:44:45.762923 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:44:45.763273 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:45.764502 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:45.765827 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:45.766648 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:45.766832 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:45.767207 (+ 375us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:44:45.767336 (+ 129us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:44:45.767419 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 03:44:45.767565 (+ 146us) server_negotiation.cc:299] Negotiation successful
0926 03:44:45.767647 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 03:44:45.771664 8807 raft_consensus.cc:1261] T b57728c01421438dac89d15fea521318 P 8b25e3df61ce40b1818bdb428b3d47a1 [term 1 FOLLOWER]: Refusing update from remote peer ed1e5938558345edaccaa79fd9ed26b9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:44:45.771880 8664 raft_consensus.cc:1261] T b57728c01421438dac89d15fea521318 P ae87bdf37fd34fd5b5dc9416b1878954 [term 1 FOLLOWER]: Refusing update from remote peer ed1e5938558345edaccaa79fd9ed26b9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:44:45.772454 8884 consensus_queue.cc:1035] T b57728c01421438dac89d15fea521318 P ed1e5938558345edaccaa79fd9ed26b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b25e3df61ce40b1818bdb428b3d47a1" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 45461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:44:45.772748 8888 consensus_queue.cc:1035] T b57728c01421438dac89d15fea521318 P ed1e5938558345edaccaa79fd9ed26b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae87bdf37fd34fd5b5dc9416b1878954" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 34427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:44:45.775533 8883 tablet_replica.cc:439] T b57728c01421438dac89d15fea521318 P 8b25e3df61ce40b1818bdb428b3d47a1: TxnStatusTablet state changed. Reason: New leader ed1e5938558345edaccaa79fd9ed26b9. Latest consensus state: current_term: 1 leader_uuid: "ed1e5938558345edaccaa79fd9ed26b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed1e5938558345edaccaa79fd9ed26b9" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 40371 } } peers { permanent_uuid: "ae87bdf37fd34fd5b5dc9416b1878954" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 34427 } } peers { permanent_uuid: "8b25e3df61ce40b1818bdb428b3d47a1" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 45461 } } }
I20260926 03:44:45.775807 8883 tablet_replica.cc:442] T b57728c01421438dac89d15fea521318 P 8b25e3df61ce40b1818bdb428b3d47a1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 03:44:45.775913 8885 tablet_replica.cc:439] T b57728c01421438dac89d15fea521318 P ae87bdf37fd34fd5b5dc9416b1878954: TxnStatusTablet state changed. Reason: New leader ed1e5938558345edaccaa79fd9ed26b9. Latest consensus state: current_term: 1 leader_uuid: "ed1e5938558345edaccaa79fd9ed26b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed1e5938558345edaccaa79fd9ed26b9" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 40371 } } peers { permanent_uuid: "ae87bdf37fd34fd5b5dc9416b1878954" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 34427 } } peers { permanent_uuid: "8b25e3df61ce40b1818bdb428b3d47a1" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 45461 } } }
I20260926 03:44:45.776248 8885 tablet_replica.cc:442] T b57728c01421438dac89d15fea521318 P ae87bdf37fd34fd5b5dc9416b1878954: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 03:44:45.778215 8888 tablet_replica.cc:439] T b57728c01421438dac89d15fea521318 P ed1e5938558345edaccaa79fd9ed26b9: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ed1e5938558345edaccaa79fd9ed26b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed1e5938558345edaccaa79fd9ed26b9" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 40371 } } peers { permanent_uuid: "ae87bdf37fd34fd5b5dc9416b1878954" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 34427 } } peers { permanent_uuid: "8b25e3df61ce40b1818bdb428b3d47a1" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 45461 } } }
I20260926 03:44:45.778496 8888 tablet_replica.cc:442] T b57728c01421438dac89d15fea521318 P ed1e5938558345edaccaa79fd9ed26b9: This TxnStatusTablet replica's current role is: LEADER
I20260926 03:44:45.778919 8892 mvcc.cc:205] Tried to move back new op lower bound from 7333454994514911232 to 7333454994374455296. Current Snapshot: MvccSnapshot[applied={T|T < 7333454994514911232}]
I20260926 03:44:45.779675 8885 tablet_replica.cc:439] T b57728c01421438dac89d15fea521318 P ae87bdf37fd34fd5b5dc9416b1878954: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ed1e5938558345edaccaa79fd9ed26b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed1e5938558345edaccaa79fd9ed26b9" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 40371 } } peers { permanent_uuid: "ae87bdf37fd34fd5b5dc9416b1878954" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 34427 } } peers { permanent_uuid: "8b25e3df61ce40b1818bdb428b3d47a1" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 45461 } } }
I20260926 03:44:45.779873 8885 tablet_replica.cc:442] T b57728c01421438dac89d15fea521318 P ae87bdf37fd34fd5b5dc9416b1878954: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 03:44:45.780169 8888 tablet_replica.cc:439] T b57728c01421438dac89d15fea521318 P ed1e5938558345edaccaa79fd9ed26b9: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ed1e5938558345edaccaa79fd9ed26b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed1e5938558345edaccaa79fd9ed26b9" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 40371 } } peers { permanent_uuid: "ae87bdf37fd34fd5b5dc9416b1878954" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 34427 } } peers { permanent_uuid: "8b25e3df61ce40b1818bdb428b3d47a1" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 45461 } } }
I20260926 03:44:45.780403 8888 tablet_replica.cc:442] T b57728c01421438dac89d15fea521318 P ed1e5938558345edaccaa79fd9ed26b9: This TxnStatusTablet replica's current role is: LEADER
I20260926 03:44:45.781487 8883 tablet_replica.cc:439] T b57728c01421438dac89d15fea521318 P 8b25e3df61ce40b1818bdb428b3d47a1: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ed1e5938558345edaccaa79fd9ed26b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed1e5938558345edaccaa79fd9ed26b9" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 40371 } } peers { permanent_uuid: "ae87bdf37fd34fd5b5dc9416b1878954" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 34427 } } peers { permanent_uuid: "8b25e3df61ce40b1818bdb428b3d47a1" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 45461 } } }
I20260926 03:44:45.781766 8883 tablet_replica.cc:442] T b57728c01421438dac89d15fea521318 P 8b25e3df61ce40b1818bdb428b3d47a1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 03:44:45.794313 8927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:45.786117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47722 (local address 127.0.194.195:45461)
0926 03:44:45.788567 (+ 2450us) server_negotiation.cc:206] Beginning negotiation
0926 03:44:45.788571 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:44:45.788589 (+ 18us) server_negotiation.cc:407] Connection header received
0926 03:44:45.788683 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:44:45.788686 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:44:45.788759 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:44:45.788890 (+ 131us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:44:45.789384 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:45.790137 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:45.791995 (+ 1858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:45.792402 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:45.792491 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:45.793009 (+ 518us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:44:45.793629 (+ 620us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:44:45.793734 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 03:44:45.793864 (+ 130us) server_negotiation.cc:299] Negotiation successful
0926 03:44:45.793926 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2338,"thread_start_us":2197,"threads_started":1}
I20260926 03:44:45.809476 8876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:45.804111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.195:54273 (local address 127.0.194.193:40371)
0926 03:44:45.804355 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0926 03:44:45.804364 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:44:45.804693 (+ 329us) server_negotiation.cc:407] Connection header received
0926 03:44:45.804896 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:44:45.804901 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:44:45.804968 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:44:45.805065 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:44:45.805540 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:45.806285 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:45.807736 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:45.808176 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:45.808320 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:45.808715 (+ 395us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:44:45.808833 (+ 118us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:44:45.808936 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0926 03:44:45.809067 (+ 131us) server_negotiation.cc:299] Negotiation successful
0926 03:44:45.809149 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 03:44:45.809541 8930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:45.804014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.193:40371 (local address 127.0.194.195:54273)
0926 03:44:45.804583 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0926 03:44:45.804599 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 03:44:45.804757 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:44:45.805086 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:44:45.805090 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:44:45.805103 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 03:44:45.805383 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:45.805391 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:45.806410 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:45.806417 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:45.807577 (+ 1160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:45.807588 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:45.808352 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:45.808359 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:45.808491 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:45.808538 (+ 47us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 03:44:45.808934 (+ 396us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 03:44:45.808946 (+ 12us) client_negotiation.cc:769] Sending connection context
0926 03:44:45.809043 (+ 97us) client_negotiation.cc:240] Negotiation successful
0926 03:44:45.809169 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":302,"threads_started":1}
I20260926 03:44:45.819470 8524 raft_consensus.cc:1261] T 1e1e4259370f4de4a4f9ec16a291796f P ed1e5938558345edaccaa79fd9ed26b9 [term 1 FOLLOWER]: Refusing update from remote peer 8b25e3df61ce40b1818bdb428b3d47a1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:44:45.820076 8664 raft_consensus.cc:1261] T 1e1e4259370f4de4a4f9ec16a291796f P ae87bdf37fd34fd5b5dc9416b1878954 [term 1 FOLLOWER]: Refusing update from remote peer 8b25e3df61ce40b1818bdb428b3d47a1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:44:45.820262 8909 consensus_queue.cc:1035] T 1e1e4259370f4de4a4f9ec16a291796f P 8b25e3df61ce40b1818bdb428b3d47a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed1e5938558345edaccaa79fd9ed26b9" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 40371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:44:45.820741 8883 consensus_queue.cc:1035] T 1e1e4259370f4de4a4f9ec16a291796f P 8b25e3df61ce40b1818bdb428b3d47a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae87bdf37fd34fd5b5dc9416b1878954" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 34427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:44:45.828269 8893 mvcc.cc:205] Tried to move back new op lower bound from 7333454994710069248 to 7333454993388466176. Current Snapshot: MvccSnapshot[applied={T|T < 7333454994710069248}]
I20260926 03:44:45.855592 8927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:45.848287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.193:33785 (local address 127.0.194.195:45461)
0926 03:44:45.848445 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0926 03:44:45.848450 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:44:45.849194 (+ 744us) server_negotiation.cc:407] Connection header received
0926 03:44:45.849400 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:44:45.849407 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:44:45.849497 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:44:45.849605 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:44:45.850346 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:45.851517 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:45.853239 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:45.853890 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:45.854017 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:45.854828 (+ 811us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:44:45.854970 (+ 142us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:44:45.855074 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0926 03:44:45.855189 (+ 115us) server_negotiation.cc:299] Negotiation successful
0926 03:44:45.855268 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 03:44:45.856098 8934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:45.848311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.195:45461 (local address 127.0.194.193:33785)
0926 03:44:45.849076 (+ 765us) negotiation.cc:107] Waiting for socket to connect
0926 03:44:45.849097 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 03:44:45.849240 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:44:45.849680 (+ 440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:44:45.849685 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:44:45.849698 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 03:44:45.850155 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:45.850170 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:45.851673 (+ 1503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:45.851677 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:45.853085 (+ 1408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:45.853100 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:45.854062 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:45.854068 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:45.854387 (+ 319us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:45.854480 (+ 93us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 03:44:45.855073 (+ 593us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 03:44:45.855086 (+ 13us) client_negotiation.cc:769] Sending connection context
0926 03:44:45.855169 (+ 83us) client_negotiation.cc:240] Negotiation successful
0926 03:44:45.855311 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":540,"thread_start_us":367,"threads_started":1}
I20260926 03:44:45.913455 8367 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56336:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:44:45.913733 8367 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56336:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:44:45.917259 8367 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f8f4c2ad1ed14a06a9cac074539f400f: Sending DeleteTablet for 3 replicas of tablet 1e1e4259370f4de4a4f9ec16a291796f
I20260926 03:44:45.918125 8786 tablet_service.cc:1514] Processing DeleteTablet for tablet 1e1e4259370f4de4a4f9ec16a291796f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:44:45 UTC) from {username='oryx', principal='oryx/127.0.194.254@KRBTEST.COM'} at 127.0.0.1:47720
I20260926 03:44:45.918267 8642 tablet_service.cc:1514] Processing DeleteTablet for tablet 1e1e4259370f4de4a4f9ec16a291796f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:44:45 UTC) from {username='oryx', principal='oryx/127.0.194.254@KRBTEST.COM'} at 127.0.0.1:45630
I20260926 03:44:45.919238 8504 tablet_service.cc:1514] Processing DeleteTablet for tablet 1e1e4259370f4de4a4f9ec16a291796f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:44:45 UTC) from {username='oryx', principal='oryx/127.0.194.254@KRBTEST.COM'} at 127.0.0.1:42844
I20260926 03:44:45.933633 8939 tablet_replica.cc:330] T 1e1e4259370f4de4a4f9ec16a291796f P ae87bdf37fd34fd5b5dc9416b1878954: stopping tablet replica
I20260926 03:44:45.933884 8939 raft_consensus.cc:2229] T 1e1e4259370f4de4a4f9ec16a291796f P ae87bdf37fd34fd5b5dc9416b1878954 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:44:45.934125 8939 raft_consensus.cc:2258] T 1e1e4259370f4de4a4f9ec16a291796f P ae87bdf37fd34fd5b5dc9416b1878954 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:44:45.935401 8939 ts_tablet_manager.cc:1905] T 1e1e4259370f4de4a4f9ec16a291796f P ae87bdf37fd34fd5b5dc9416b1878954: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:44:45.942682 8939 ts_tablet_manager.cc:1918] T 1e1e4259370f4de4a4f9ec16a291796f P ae87bdf37fd34fd5b5dc9416b1878954: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 03:44:45.942842 8939 log.cc:1196] T 1e1e4259370f4de4a4f9ec16a291796f P ae87bdf37fd34fd5b5dc9416b1878954: Deleting WAL directory at /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790394221564982-779-0/minicluster-data/ts-1/wal/wals/1e1e4259370f4de4a4f9ec16a291796f
I20260926 03:44:45.943300 8939 ts_tablet_manager.cc:1939] T 1e1e4259370f4de4a4f9ec16a291796f P ae87bdf37fd34fd5b5dc9416b1878954: Deleting consensus metadata
I20260926 03:44:45.944475 8353 catalog_manager.cc:4841] TS ae87bdf37fd34fd5b5dc9416b1878954 (127.0.194.194:34427): tablet 1e1e4259370f4de4a4f9ec16a291796f (table test-table [id=a79aecd0cd0a46b9b31299a6a897e34b]) successfully deleted
I20260926 03:44:45.951136 8940 tablet_replica.cc:330] T 1e1e4259370f4de4a4f9ec16a291796f P ed1e5938558345edaccaa79fd9ed26b9: stopping tablet replica
I20260926 03:44:45.951472 8940 raft_consensus.cc:2229] T 1e1e4259370f4de4a4f9ec16a291796f P ed1e5938558345edaccaa79fd9ed26b9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:44:45.951951 8940 raft_consensus.cc:2258] T 1e1e4259370f4de4a4f9ec16a291796f P ed1e5938558345edaccaa79fd9ed26b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:44:45.960918 8938 tablet_replica.cc:330] T 1e1e4259370f4de4a4f9ec16a291796f P 8b25e3df61ce40b1818bdb428b3d47a1: stopping tablet replica
I20260926 03:44:45.962270 8940 ts_tablet_manager.cc:1905] T 1e1e4259370f4de4a4f9ec16a291796f P ed1e5938558345edaccaa79fd9ed26b9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:44:45.962586 8938 raft_consensus.cc:2229] T 1e1e4259370f4de4a4f9ec16a291796f P 8b25e3df61ce40b1818bdb428b3d47a1 [term 1 LEADER]: Raft consensus shutting down.
I20260926 03:44:45.964293 8938 raft_consensus.cc:2258] T 1e1e4259370f4de4a4f9ec16a291796f P 8b25e3df61ce40b1818bdb428b3d47a1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:44:45.965864 8938 ts_tablet_manager.cc:1905] T 1e1e4259370f4de4a4f9ec16a291796f P 8b25e3df61ce40b1818bdb428b3d47a1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:44:45.978190 8940 ts_tablet_manager.cc:1918] T 1e1e4259370f4de4a4f9ec16a291796f P ed1e5938558345edaccaa79fd9ed26b9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
Sep 26 03:44:45 dist-test-slave-n2th krb5kdc[8322](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394285, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
I20260926 03:44:45.990868 8940 log.cc:1196] T 1e1e4259370f4de4a4f9ec16a291796f P ed1e5938558345edaccaa79fd9ed26b9: Deleting WAL directory at /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790394221564982-779-0/minicluster-data/ts-0/wal/wals/1e1e4259370f4de4a4f9ec16a291796f
I20260926 03:44:45.991535 8940 ts_tablet_manager.cc:1939] T 1e1e4259370f4de4a4f9ec16a291796f P ed1e5938558345edaccaa79fd9ed26b9: Deleting consensus metadata
I20260926 03:44:45.991948 8938 ts_tablet_manager.cc:1918] T 1e1e4259370f4de4a4f9ec16a291796f P 8b25e3df61ce40b1818bdb428b3d47a1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 03:44:45.993263 8354 catalog_manager.cc:4841] TS ed1e5938558345edaccaa79fd9ed26b9 (127.0.194.193:40371): tablet 1e1e4259370f4de4a4f9ec16a291796f (table test-table [id=a79aecd0cd0a46b9b31299a6a897e34b]) successfully deleted
I20260926 03:44:45.993624 8938 log.cc:1196] T 1e1e4259370f4de4a4f9ec16a291796f P 8b25e3df61ce40b1818bdb428b3d47a1: Deleting WAL directory at /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790394221564982-779-0/minicluster-data/ts-2/wal/wals/1e1e4259370f4de4a4f9ec16a291796f
I20260926 03:44:45.994230 8938 ts_tablet_manager.cc:1939] T 1e1e4259370f4de4a4f9ec16a291796f P 8b25e3df61ce40b1818bdb428b3d47a1: Deleting consensus metadata
I20260926 03:44:45.999641 8351 catalog_manager.cc:4841] TS 8b25e3df61ce40b1818bdb428b3d47a1 (127.0.194.195:45461): tablet 1e1e4259370f4de4a4f9ec16a291796f (table test-table [id=a79aecd0cd0a46b9b31299a6a897e34b]) successfully deleted
Sep 26 03:44:46 dist-test-slave-n2th krb5kdc[8322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.194.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:44:46 dist-test-slave-n2th krb5kdc[8322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.194.254@KRBTEST.COM, Server not found in Kerberos database
W20260926 03:44:46.030086 8462 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:44:46.016353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56352 (local address 127.0.194.254:34071)
0926 03:44:46.016564 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0926 03:44:46.016570 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:44:46.017212 (+ 642us) server_negotiation.cc:407] Connection header received
0926 03:44:46.017402 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:44:46.017408 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:44:46.017498 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:44:46.017610 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:44:46.019312 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:46.020465 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:46.022275 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:46.023003 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:46.023119 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:46.029800 (+ 6681us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56352: BlockingRecv error: recv got EOF from 127.0.0.1:56352 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Sep 26 03:44:46 dist-test-slave-n2th krb5kdc[8322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394285, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.194.254@KRBTEST.COM
I20260926 03:44:46.052351 8462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:46.034089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56360 (local address 127.0.194.254:34071)
0926 03:44:46.034257 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0926 03:44:46.034265 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:44:46.035010 (+ 745us) server_negotiation.cc:407] Connection header received
0926 03:44:46.035170 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:44:46.035176 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:44:46.035266 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:44:46.035371 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:44:46.036887 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:46.038041 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:46.039931 (+ 1890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:46.040578 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:46.040708 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:46.045656 (+ 4948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:44:46.045663 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:44:46.045668 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:44:46.045741 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:44:46.049436 (+ 3695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:46.050273 (+ 837us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:46.050279 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:46.050283 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:46.050437 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:46.050838 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:46.050844 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:46.050849 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:46.051201 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:44:46.051445 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0926 03:44:46.051766 (+ 321us) server_negotiation.cc:299] Negotiation successful
0926 03:44:46.051998 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 03:44:46.056529 8367 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56360:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 03:44:46.057228 8367 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 03:44:46.064886 8504 tablet_service.cc:1467] Processing CreateTablet for tablet bc9ccfe6b76d48cc9cc23854038db3c6 (DEFAULT_TABLE table=test-table [id=c5fd031946c44a5b9e2dd984a0772aee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:44:46.065377 8504 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc9ccfe6b76d48cc9cc23854038db3c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:44:46.065513 8642 tablet_service.cc:1467] Processing CreateTablet for tablet bc9ccfe6b76d48cc9cc23854038db3c6 (DEFAULT_TABLE table=test-table [id=c5fd031946c44a5b9e2dd984a0772aee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:44:46.065970 8642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc9ccfe6b76d48cc9cc23854038db3c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:44:46.068974 8786 tablet_service.cc:1467] Processing CreateTablet for tablet bc9ccfe6b76d48cc9cc23854038db3c6 (DEFAULT_TABLE table=test-table [id=c5fd031946c44a5b9e2dd984a0772aee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:44:46.069391 8786 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc9ccfe6b76d48cc9cc23854038db3c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:44:46.072461 8878 tablet_bootstrap.cc:492] T bc9ccfe6b76d48cc9cc23854038db3c6 P ed1e5938558345edaccaa79fd9ed26b9: Bootstrap starting.
I20260926 03:44:46.074115 8878 tablet_bootstrap.cc:654] T bc9ccfe6b76d48cc9cc23854038db3c6 P ed1e5938558345edaccaa79fd9ed26b9: Neither blocks nor log segments found. Creating new log.
I20260926 03:44:46.082865 8877 tablet_bootstrap.cc:492] T bc9ccfe6b76d48cc9cc23854038db3c6 P 8b25e3df61ce40b1818bdb428b3d47a1: Bootstrap starting.
I20260926 03:44:46.083333 8879 tablet_bootstrap.cc:492] T bc9ccfe6b76d48cc9cc23854038db3c6 P ae87bdf37fd34fd5b5dc9416b1878954: Bootstrap starting.
I20260926 03:44:46.084647 8877 tablet_bootstrap.cc:654] T bc9ccfe6b76d48cc9cc23854038db3c6 P 8b25e3df61ce40b1818bdb428b3d47a1: Neither blocks nor log segments found. Creating new log.
I20260926 03:44:46.084940 8879 tablet_bootstrap.cc:654] T bc9ccfe6b76d48cc9cc23854038db3c6 P ae87bdf37fd34fd5b5dc9416b1878954: Neither blocks nor log segments found. Creating new log.
I20260926 03:44:46.092008 8879 tablet_bootstrap.cc:492] T bc9ccfe6b76d48cc9cc23854038db3c6 P ae87bdf37fd34fd5b5dc9416b1878954: No bootstrap required, opened a new log
I20260926 03:44:46.092248 8879 ts_tablet_manager.cc:1397] T bc9ccfe6b76d48cc9cc23854038db3c6 P ae87bdf37fd34fd5b5dc9416b1878954: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260926 03:44:46.093102 8879 raft_consensus.cc:348] T bc9ccfe6b76d48cc9cc23854038db3c6 P ae87bdf37fd34fd5b5dc9416b1878954 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed1e5938558345edaccaa79fd9ed26b9" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 40371 } } peers { permanent_uuid: "ae87bdf37fd34fd5b5dc9416b1878954" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 34427 } } peers { permanent_uuid: "8b25e3df61ce40b1818bdb428b3d47a1" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 45461 } }
I20260926 03:44:46.093508 8879 raft_consensus.cc:374] T bc9ccfe6b76d48cc9cc23854038db3c6 P ae87bdf37fd34fd5b5dc9416b1878954 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:44:46.093659 8879 raft_consensus.cc:729] T bc9ccfe6b76d48cc9cc23854038db3c6 P ae87bdf37fd34fd5b5dc9416b1878954 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae87bdf37fd34fd5b5dc9416b1878954, State: Initialized, Role: FOLLOWER
I20260926 03:44:46.094053 8879 consensus_queue.cc:260] T bc9ccfe6b76d48cc9cc23854038db3c6 P ae87bdf37fd34fd5b5dc9416b1878954 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed1e5938558345edaccaa79fd9ed26b9" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 40371 } } peers { permanent_uuid: "ae87bdf37fd34fd5b5dc9416b1878954" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 34427 } } peers { permanent_uuid: "8b25e3df61ce40b1818bdb428b3d47a1" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 45461 } }
I20260926 03:44:46.094764 8879 ts_tablet_manager.cc:1428] T bc9ccfe6b76d48cc9cc23854038db3c6 P ae87bdf37fd34fd5b5dc9416b1878954: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260926 03:44:46.102478 8877 tablet_bootstrap.cc:492] T bc9ccfe6b76d48cc9cc23854038db3c6 P 8b25e3df61ce40b1818bdb428b3d47a1: No bootstrap required, opened a new log
I20260926 03:44:46.102663 8877 ts_tablet_manager.cc:1397] T bc9ccfe6b76d48cc9cc23854038db3c6 P 8b25e3df61ce40b1818bdb428b3d47a1: Time spent bootstrapping tablet: real 0.020s user 0.002s sys 0.007s
I20260926 03:44:46.103487 8877 raft_consensus.cc:348] T bc9ccfe6b76d48cc9cc23854038db3c6 P 8b25e3df61ce40b1818bdb428b3d47a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed1e5938558345edaccaa79fd9ed26b9" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 40371 } } peers { permanent_uuid: "ae87bdf37fd34fd5b5dc9416b1878954" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 34427 } } peers { permanent_uuid: "8b25e3df61ce40b1818bdb428b3d47a1" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 45461 } }
I20260926 03:44:46.103790 8877 raft_consensus.cc:374] T bc9ccfe6b76d48cc9cc23854038db3c6 P 8b25e3df61ce40b1818bdb428b3d47a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:44:46.103894 8877 raft_consensus.cc:729] T bc9ccfe6b76d48cc9cc23854038db3c6 P 8b25e3df61ce40b1818bdb428b3d47a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b25e3df61ce40b1818bdb428b3d47a1, State: Initialized, Role: FOLLOWER
I20260926 03:44:46.104336 8877 consensus_queue.cc:260] T bc9ccfe6b76d48cc9cc23854038db3c6 P 8b25e3df61ce40b1818bdb428b3d47a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed1e5938558345edaccaa79fd9ed26b9" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 40371 } } peers { permanent_uuid: "ae87bdf37fd34fd5b5dc9416b1878954" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 34427 } } peers { permanent_uuid: "8b25e3df61ce40b1818bdb428b3d47a1" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 45461 } }
I20260926 03:44:46.105726 8877 ts_tablet_manager.cc:1428] T bc9ccfe6b76d48cc9cc23854038db3c6 P 8b25e3df61ce40b1818bdb428b3d47a1: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260926 03:44:46.110977 8878 tablet_bootstrap.cc:492] T bc9ccfe6b76d48cc9cc23854038db3c6 P ed1e5938558345edaccaa79fd9ed26b9: No bootstrap required, opened a new log
I20260926 03:44:46.111231 8878 ts_tablet_manager.cc:1397] T bc9ccfe6b76d48cc9cc23854038db3c6 P ed1e5938558345edaccaa79fd9ed26b9: Time spent bootstrapping tablet: real 0.039s user 0.014s sys 0.002s
I20260926 03:44:46.112077 8878 raft_consensus.cc:348] T bc9ccfe6b76d48cc9cc23854038db3c6 P ed1e5938558345edaccaa79fd9ed26b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed1e5938558345edaccaa79fd9ed26b9" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 40371 } } peers { permanent_uuid: "ae87bdf37fd34fd5b5dc9416b1878954" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 34427 } } peers { permanent_uuid: "8b25e3df61ce40b1818bdb428b3d47a1" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 45461 } }
I20260926 03:44:46.112355 8878 raft_consensus.cc:374] T bc9ccfe6b76d48cc9cc23854038db3c6 P ed1e5938558345edaccaa79fd9ed26b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:44:46.112457 8878 raft_consensus.cc:729] T bc9ccfe6b76d48cc9cc23854038db3c6 P ed1e5938558345edaccaa79fd9ed26b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed1e5938558345edaccaa79fd9ed26b9, State: Initialized, Role: FOLLOWER
I20260926 03:44:46.112722 8878 consensus_queue.cc:260] T bc9ccfe6b76d48cc9cc23854038db3c6 P ed1e5938558345edaccaa79fd9ed26b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed1e5938558345edaccaa79fd9ed26b9" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 40371 } } peers { permanent_uuid: "ae87bdf37fd34fd5b5dc9416b1878954" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 34427 } } peers { permanent_uuid: "8b25e3df61ce40b1818bdb428b3d47a1" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 45461 } }
I20260926 03:44:46.113432 8878 ts_tablet_manager.cc:1428] T bc9ccfe6b76d48cc9cc23854038db3c6 P ed1e5938558345edaccaa79fd9ed26b9: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260926 03:44:46.216281 8896 raft_consensus.cc:482] T bc9ccfe6b76d48cc9cc23854038db3c6 P ed1e5938558345edaccaa79fd9ed26b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 03:44:46.216522 8896 raft_consensus.cc:504] T bc9ccfe6b76d48cc9cc23854038db3c6 P ed1e5938558345edaccaa79fd9ed26b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed1e5938558345edaccaa79fd9ed26b9" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 40371 } } peers { permanent_uuid: "ae87bdf37fd34fd5b5dc9416b1878954" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 34427 } } peers { permanent_uuid: "8b25e3df61ce40b1818bdb428b3d47a1" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 45461 } }
I20260926 03:44:46.217300 8896 leader_election.cc:288] T bc9ccfe6b76d48cc9cc23854038db3c6 P ed1e5938558345edaccaa79fd9ed26b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ae87bdf37fd34fd5b5dc9416b1878954 (127.0.194.194:34427), 8b25e3df61ce40b1818bdb428b3d47a1 (127.0.194.195:45461)
I20260926 03:44:46.217875 8664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc9ccfe6b76d48cc9cc23854038db3c6" candidate_uuid: "ed1e5938558345edaccaa79fd9ed26b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae87bdf37fd34fd5b5dc9416b1878954" is_pre_election: true
I20260926 03:44:46.217986 8807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc9ccfe6b76d48cc9cc23854038db3c6" candidate_uuid: "ed1e5938558345edaccaa79fd9ed26b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b25e3df61ce40b1818bdb428b3d47a1" is_pre_election: true
I20260926 03:44:46.218128 8664 raft_consensus.cc:2454] T bc9ccfe6b76d48cc9cc23854038db3c6 P ae87bdf37fd34fd5b5dc9416b1878954 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed1e5938558345edaccaa79fd9ed26b9 in term 0.
I20260926 03:44:46.218218 8807 raft_consensus.cc:2454] T bc9ccfe6b76d48cc9cc23854038db3c6 P 8b25e3df61ce40b1818bdb428b3d47a1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed1e5938558345edaccaa79fd9ed26b9 in term 0.
I20260926 03:44:46.218645 8456 leader_election.cc:302] T bc9ccfe6b76d48cc9cc23854038db3c6 P ed1e5938558345edaccaa79fd9ed26b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae87bdf37fd34fd5b5dc9416b1878954, ed1e5938558345edaccaa79fd9ed26b9; no voters:
I20260926 03:44:46.218971 8896 raft_consensus.cc:2781] T bc9ccfe6b76d48cc9cc23854038db3c6 P ed1e5938558345edaccaa79fd9ed26b9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 03:44:46.219094 8896 raft_consensus.cc:482] T bc9ccfe6b76d48cc9cc23854038db3c6 P ed1e5938558345edaccaa79fd9ed26b9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 03:44:46.219182 8896 raft_consensus.cc:3037] T bc9ccfe6b76d48cc9cc23854038db3c6 P ed1e5938558345edaccaa79fd9ed26b9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:44:46.222419 8896 raft_consensus.cc:504] T bc9ccfe6b76d48cc9cc23854038db3c6 P ed1e5938558345edaccaa79fd9ed26b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed1e5938558345edaccaa79fd9ed26b9" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 40371 } } peers { permanent_uuid: "ae87bdf37fd34fd5b5dc9416b1878954" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 34427 } } peers { permanent_uuid: "8b25e3df61ce40b1818bdb428b3d47a1" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 45461 } }
I20260926 03:44:46.223544 8664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc9ccfe6b76d48cc9cc23854038db3c6" candidate_uuid: "ed1e5938558345edaccaa79fd9ed26b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae87bdf37fd34fd5b5dc9416b1878954"
I20260926 03:44:46.223780 8664 raft_consensus.cc:3037] T bc9ccfe6b76d48cc9cc23854038db3c6 P ae87bdf37fd34fd5b5dc9416b1878954 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:44:46.226794 8664 raft_consensus.cc:2454] T bc9ccfe6b76d48cc9cc23854038db3c6 P ae87bdf37fd34fd5b5dc9416b1878954 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed1e5938558345edaccaa79fd9ed26b9 in term 1.
I20260926 03:44:46.227319 8456 leader_election.cc:302] T bc9ccfe6b76d48cc9cc23854038db3c6 P ed1e5938558345edaccaa79fd9ed26b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae87bdf37fd34fd5b5dc9416b1878954, ed1e5938558345edaccaa79fd9ed26b9; no voters:
I20260926 03:44:46.227641 8888 raft_consensus.cc:2781] T bc9ccfe6b76d48cc9cc23854038db3c6 P ed1e5938558345edaccaa79fd9ed26b9 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:44:46.227799 8888 raft_consensus.cc:686] T bc9ccfe6b76d48cc9cc23854038db3c6 P ed1e5938558345edaccaa79fd9ed26b9 [term 1 LEADER]: Becoming Leader. State: Replica: ed1e5938558345edaccaa79fd9ed26b9, State: Running, Role: LEADER
I20260926 03:44:46.228078 8888 consensus_queue.cc:237] T bc9ccfe6b76d48cc9cc23854038db3c6 P ed1e5938558345edaccaa79fd9ed26b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed1e5938558345edaccaa79fd9ed26b9" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 40371 } } peers { permanent_uuid: "ae87bdf37fd34fd5b5dc9416b1878954" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 34427 } } peers { permanent_uuid: "8b25e3df61ce40b1818bdb428b3d47a1" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 45461 } }
I20260926 03:44:46.230324 8896 leader_election.cc:288] T bc9ccfe6b76d48cc9cc23854038db3c6 P ed1e5938558345edaccaa79fd9ed26b9 [CANDIDATE]: Term 1 election: Requested vote from peers ae87bdf37fd34fd5b5dc9416b1878954 (127.0.194.194:34427), 8b25e3df61ce40b1818bdb428b3d47a1 (127.0.194.195:45461)
I20260926 03:44:46.230952 8807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc9ccfe6b76d48cc9cc23854038db3c6" candidate_uuid: "ed1e5938558345edaccaa79fd9ed26b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b25e3df61ce40b1818bdb428b3d47a1"
I20260926 03:44:46.231189 8807 raft_consensus.cc:3037] T bc9ccfe6b76d48cc9cc23854038db3c6 P 8b25e3df61ce40b1818bdb428b3d47a1 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:44:46.234382 8807 raft_consensus.cc:2454] T bc9ccfe6b76d48cc9cc23854038db3c6 P 8b25e3df61ce40b1818bdb428b3d47a1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed1e5938558345edaccaa79fd9ed26b9 in term 1.
I20260926 03:44:46.240284 8367 catalog_manager.cc:5495] T bc9ccfe6b76d48cc9cc23854038db3c6 P ed1e5938558345edaccaa79fd9ed26b9 reported cstate change: term changed from 0 to 1, leader changed from <none> to ed1e5938558345edaccaa79fd9ed26b9 (127.0.194.193). New cstate: current_term: 1 leader_uuid: "ed1e5938558345edaccaa79fd9ed26b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed1e5938558345edaccaa79fd9ed26b9" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 40371 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae87bdf37fd34fd5b5dc9416b1878954" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 34427 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b25e3df61ce40b1818bdb428b3d47a1" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 45461 } health_report { overall_health: UNKNOWN } } }
I20260926 03:44:46.292528 8876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:46.285901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42878 (local address 127.0.194.193:40371)
0926 03:44:46.286086 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0926 03:44:46.286094 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:44:46.286113 (+ 19us) server_negotiation.cc:407] Connection header received
0926 03:44:46.286327 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:44:46.286336 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:44:46.286429 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:44:46.286531 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:44:46.287061 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:46.288279 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:46.290313 (+ 2034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:46.291055 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:46.291194 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:46.291650 (+ 456us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:44:46.291806 (+ 156us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:44:46.291902 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0926 03:44:46.292079 (+ 177us) server_negotiation.cc:299] Negotiation successful
0926 03:44:46.292174 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 03:44:46.295512 8664 raft_consensus.cc:1261] T bc9ccfe6b76d48cc9cc23854038db3c6 P ae87bdf37fd34fd5b5dc9416b1878954 [term 1 FOLLOWER]: Refusing update from remote peer ed1e5938558345edaccaa79fd9ed26b9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:44:46.296185 8807 raft_consensus.cc:1261] T bc9ccfe6b76d48cc9cc23854038db3c6 P 8b25e3df61ce40b1818bdb428b3d47a1 [term 1 FOLLOWER]: Refusing update from remote peer ed1e5938558345edaccaa79fd9ed26b9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:44:46.296240 8888 consensus_queue.cc:1035] T bc9ccfe6b76d48cc9cc23854038db3c6 P ed1e5938558345edaccaa79fd9ed26b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae87bdf37fd34fd5b5dc9416b1878954" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 34427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:44:46.297253 8888 consensus_queue.cc:1035] T bc9ccfe6b76d48cc9cc23854038db3c6 P ed1e5938558345edaccaa79fd9ed26b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b25e3df61ce40b1818bdb428b3d47a1" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 45461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:44:46.307766 8366 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56360:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:44:46.307998 8366 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56360:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:44:46.311115 8366 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f8f4c2ad1ed14a06a9cac074539f400f: Sending DeleteTablet for 3 replicas of tablet bc9ccfe6b76d48cc9cc23854038db3c6
I20260926 03:44:46.311946 8504 tablet_service.cc:1514] Processing DeleteTablet for tablet bc9ccfe6b76d48cc9cc23854038db3c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:44:46 UTC) from {username='oryx', principal='oryx/127.0.194.254@KRBTEST.COM'} at 127.0.0.1:42844
I20260926 03:44:46.312053 8786 tablet_service.cc:1514] Processing DeleteTablet for tablet bc9ccfe6b76d48cc9cc23854038db3c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:44:46 UTC) from {username='oryx', principal='oryx/127.0.194.254@KRBTEST.COM'} at 127.0.0.1:47720
I20260926 03:44:46.312124 8642 tablet_service.cc:1514] Processing DeleteTablet for tablet bc9ccfe6b76d48cc9cc23854038db3c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:44:46 UTC) from {username='oryx', principal='oryx/127.0.194.254@KRBTEST.COM'} at 127.0.0.1:45630
I20260926 03:44:46.312381 8939 tablet_replica.cc:330] T bc9ccfe6b76d48cc9cc23854038db3c6 P ae87bdf37fd34fd5b5dc9416b1878954: stopping tablet replica
I20260926 03:44:46.312399 8940 tablet_replica.cc:330] T bc9ccfe6b76d48cc9cc23854038db3c6 P ed1e5938558345edaccaa79fd9ed26b9: stopping tablet replica
I20260926 03:44:46.312589 8940 raft_consensus.cc:2229] T bc9ccfe6b76d48cc9cc23854038db3c6 P ed1e5938558345edaccaa79fd9ed26b9 [term 1 LEADER]: Raft consensus shutting down.
I20260926 03:44:46.312590 8939 raft_consensus.cc:2229] T bc9ccfe6b76d48cc9cc23854038db3c6 P ae87bdf37fd34fd5b5dc9416b1878954 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:44:46.312803 8939 raft_consensus.cc:2258] T bc9ccfe6b76d48cc9cc23854038db3c6 P ae87bdf37fd34fd5b5dc9416b1878954 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:44:46.312888 8940 raft_consensus.cc:2258] T bc9ccfe6b76d48cc9cc23854038db3c6 P ed1e5938558345edaccaa79fd9ed26b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:44:46.313194 8938 tablet_replica.cc:330] T bc9ccfe6b76d48cc9cc23854038db3c6 P 8b25e3df61ce40b1818bdb428b3d47a1: stopping tablet replica
I20260926 03:44:46.313923 8939 ts_tablet_manager.cc:1905] T bc9ccfe6b76d48cc9cc23854038db3c6 P ae87bdf37fd34fd5b5dc9416b1878954: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:44:46.314626 8940 ts_tablet_manager.cc:1905] T bc9ccfe6b76d48cc9cc23854038db3c6 P ed1e5938558345edaccaa79fd9ed26b9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:44:46.316730 8938 raft_consensus.cc:2229] T bc9ccfe6b76d48cc9cc23854038db3c6 P 8b25e3df61ce40b1818bdb428b3d47a1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:44:46.317240 8938 pending_rounds.cc:62] T bc9ccfe6b76d48cc9cc23854038db3c6 P 8b25e3df61ce40b1818bdb428b3d47a1: Trying to abort 2 pending ops.
I20260926 03:44:46.317363 8938 pending_rounds.cc:68] T bc9ccfe6b76d48cc9cc23854038db3c6 P 8b25e3df61ce40b1818bdb428b3d47a1: Aborting op as it isn't in flight: id { term: 1 index: 2 } timestamp: 7333454996660121600 op_type: WRITE_OP write_request { tablet_id: "bc9ccfe6b76d48cc9cc23854038db3c6" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "val" type: INT32 is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\003\00090\000\0001\324\000\000" indirect_data: "<redacted>""" } external_consistency_mode: CLIENT_PROPAGATED authz_token { token_data: "\010\332\201\335\325\006\"8\n\ntest-admin\022*\n c5fd031946c44a5b9e2dd984a0772aee\020\001\030\001 \001(\001" signature: "<redacted>""]W\220e\200\247\013\3753 sd\303\245Z\031+\327\2636$g\313-\302\177h\250\253\241\342\214\005\324&r\003\343\344\317#\257Qf\271G\345u\327\3616\245;\n\215\362\246\375<\363\t\010\234\n" signing_key_seq_num: 0 } } request_id { client_id: "29f64fd8e1964fd5ad6f7b8ea3f9e750" seq_no: 0 first_incomplete_seq_no: 0 attempt_no: 0 }
I20260926 03:44:46.318113 8938 pending_rounds.cc:68] T bc9ccfe6b76d48cc9cc23854038db3c6 P 8b25e3df61ce40b1818bdb428b3d47a1: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7333454996392914944 op_type: NO_OP noop_request { }
I20260926 03:44:46.318284 8938 raft_consensus.cc:2866] T bc9ccfe6b76d48cc9cc23854038db3c6 P 8b25e3df61ce40b1818bdb428b3d47a1 [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260926 03:44:46.318418 8938 raft_consensus.cc:2258] T bc9ccfe6b76d48cc9cc23854038db3c6 P 8b25e3df61ce40b1818bdb428b3d47a1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:44:46.320674 8939 ts_tablet_manager.cc:1918] T bc9ccfe6b76d48cc9cc23854038db3c6 P ae87bdf37fd34fd5b5dc9416b1878954: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:44:46.320824 8939 log.cc:1196] T bc9ccfe6b76d48cc9cc23854038db3c6 P ae87bdf37fd34fd5b5dc9416b1878954: Deleting WAL directory at /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790394221564982-779-0/minicluster-data/ts-1/wal/wals/bc9ccfe6b76d48cc9cc23854038db3c6
I20260926 03:44:46.321251 8939 ts_tablet_manager.cc:1939] T bc9ccfe6b76d48cc9cc23854038db3c6 P ae87bdf37fd34fd5b5dc9416b1878954: Deleting consensus metadata
I20260926 03:44:46.321791 8938 ts_tablet_manager.cc:1905] T bc9ccfe6b76d48cc9cc23854038db3c6 P 8b25e3df61ce40b1818bdb428b3d47a1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:44:46.322306 8353 catalog_manager.cc:4841] TS ae87bdf37fd34fd5b5dc9416b1878954 (127.0.194.194:34427): tablet bc9ccfe6b76d48cc9cc23854038db3c6 (table test-table [id=c5fd031946c44a5b9e2dd984a0772aee]) successfully deleted
I20260926 03:44:46.323164 8940 ts_tablet_manager.cc:1918] T bc9ccfe6b76d48cc9cc23854038db3c6 P ed1e5938558345edaccaa79fd9ed26b9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:44:46.323312 8940 log.cc:1196] T bc9ccfe6b76d48cc9cc23854038db3c6 P ed1e5938558345edaccaa79fd9ed26b9: Deleting WAL directory at /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790394221564982-779-0/minicluster-data/ts-0/wal/wals/bc9ccfe6b76d48cc9cc23854038db3c6
I20260926 03:44:46.323714 8940 ts_tablet_manager.cc:1939] T bc9ccfe6b76d48cc9cc23854038db3c6 P ed1e5938558345edaccaa79fd9ed26b9: Deleting consensus metadata
I20260926 03:44:46.324837 8354 catalog_manager.cc:4841] TS ed1e5938558345edaccaa79fd9ed26b9 (127.0.194.193:40371): tablet bc9ccfe6b76d48cc9cc23854038db3c6 (table test-table [id=c5fd031946c44a5b9e2dd984a0772aee]) successfully deleted
I20260926 03:44:46.327464 8367 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56360:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 03:44:46.328409 8367 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 03:44:46.328960 8938 ts_tablet_manager.cc:1918] T bc9ccfe6b76d48cc9cc23854038db3c6 P 8b25e3df61ce40b1818bdb428b3d47a1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:44:46.329190 8938 log.cc:1196] T bc9ccfe6b76d48cc9cc23854038db3c6 P 8b25e3df61ce40b1818bdb428b3d47a1: Deleting WAL directory at /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790394221564982-779-0/minicluster-data/ts-2/wal/wals/bc9ccfe6b76d48cc9cc23854038db3c6
I20260926 03:44:46.329707 8938 ts_tablet_manager.cc:1939] T bc9ccfe6b76d48cc9cc23854038db3c6 P 8b25e3df61ce40b1818bdb428b3d47a1: Deleting consensus metadata
I20260926 03:44:46.330948 8351 catalog_manager.cc:4841] TS 8b25e3df61ce40b1818bdb428b3d47a1 (127.0.194.195:45461): tablet bc9ccfe6b76d48cc9cc23854038db3c6 (table test-table [id=c5fd031946c44a5b9e2dd984a0772aee]) successfully deleted
I20260926 03:44:46.336615 8504 tablet_service.cc:1467] Processing CreateTablet for tablet c1a5104b5d1245b89d6e3a98e505ddb0 (DEFAULT_TABLE table=test-table [id=ae36445423c64444bb1c74a38ebbb310]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:44:46.336727 8786 tablet_service.cc:1467] Processing CreateTablet for tablet c1a5104b5d1245b89d6e3a98e505ddb0 (DEFAULT_TABLE table=test-table [id=ae36445423c64444bb1c74a38ebbb310]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:44:46.336982 8642 tablet_service.cc:1467] Processing CreateTablet for tablet c1a5104b5d1245b89d6e3a98e505ddb0 (DEFAULT_TABLE table=test-table [id=ae36445423c64444bb1c74a38ebbb310]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:44:46.337124 8504 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1a5104b5d1245b89d6e3a98e505ddb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:44:46.337194 8786 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1a5104b5d1245b89d6e3a98e505ddb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:44:46.337363 8642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1a5104b5d1245b89d6e3a98e505ddb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:44:46.343978 8878 tablet_bootstrap.cc:492] T c1a5104b5d1245b89d6e3a98e505ddb0 P ed1e5938558345edaccaa79fd9ed26b9: Bootstrap starting.
I20260926 03:44:46.345779 8878 tablet_bootstrap.cc:654] T c1a5104b5d1245b89d6e3a98e505ddb0 P ed1e5938558345edaccaa79fd9ed26b9: Neither blocks nor log segments found. Creating new log.
I20260926 03:44:46.346949 8877 tablet_bootstrap.cc:492] T c1a5104b5d1245b89d6e3a98e505ddb0 P 8b25e3df61ce40b1818bdb428b3d47a1: Bootstrap starting.
I20260926 03:44:46.348726 8877 tablet_bootstrap.cc:654] T c1a5104b5d1245b89d6e3a98e505ddb0 P 8b25e3df61ce40b1818bdb428b3d47a1: Neither blocks nor log segments found. Creating new log.
I20260926 03:44:46.352717 8879 tablet_bootstrap.cc:492] T c1a5104b5d1245b89d6e3a98e505ddb0 P ae87bdf37fd34fd5b5dc9416b1878954: Bootstrap starting.
I20260926 03:44:46.354451 8879 tablet_bootstrap.cc:654] T c1a5104b5d1245b89d6e3a98e505ddb0 P ae87bdf37fd34fd5b5dc9416b1878954: Neither blocks nor log segments found. Creating new log.
I20260926 03:44:46.358028 8878 tablet_bootstrap.cc:492] T c1a5104b5d1245b89d6e3a98e505ddb0 P ed1e5938558345edaccaa79fd9ed26b9: No bootstrap required, opened a new log
I20260926 03:44:46.358315 8878 ts_tablet_manager.cc:1397] T c1a5104b5d1245b89d6e3a98e505ddb0 P ed1e5938558345edaccaa79fd9ed26b9: Time spent bootstrapping tablet: real 0.014s user 0.001s sys 0.002s
I20260926 03:44:46.360809 8878 raft_consensus.cc:348] T c1a5104b5d1245b89d6e3a98e505ddb0 P ed1e5938558345edaccaa79fd9ed26b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b25e3df61ce40b1818bdb428b3d47a1" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 45461 } } peers { permanent_uuid: "ed1e5938558345edaccaa79fd9ed26b9" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 40371 } } peers { permanent_uuid: "ae87bdf37fd34fd5b5dc9416b1878954" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 34427 } }
I20260926 03:44:46.361189 8878 raft_consensus.cc:374] T c1a5104b5d1245b89d6e3a98e505ddb0 P ed1e5938558345edaccaa79fd9ed26b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:44:46.361565 8878 raft_consensus.cc:729] T c1a5104b5d1245b89d6e3a98e505ddb0 P ed1e5938558345edaccaa79fd9ed26b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed1e5938558345edaccaa79fd9ed26b9, State: Initialized, Role: FOLLOWER
I20260926 03:44:46.361923 8878 consensus_queue.cc:260] T c1a5104b5d1245b89d6e3a98e505ddb0 P ed1e5938558345edaccaa79fd9ed26b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b25e3df61ce40b1818bdb428b3d47a1" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 45461 } } peers { permanent_uuid: "ed1e5938558345edaccaa79fd9ed26b9" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 40371 } } peers { permanent_uuid: "ae87bdf37fd34fd5b5dc9416b1878954" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 34427 } }
I20260926 03:44:46.362614 8877 tablet_bootstrap.cc:492] T c1a5104b5d1245b89d6e3a98e505ddb0 P 8b25e3df61ce40b1818bdb428b3d47a1: No bootstrap required, opened a new log
I20260926 03:44:46.362856 8877 ts_tablet_manager.cc:1397] T c1a5104b5d1245b89d6e3a98e505ddb0 P 8b25e3df61ce40b1818bdb428b3d47a1: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.002s
I20260926 03:44:46.363483 8878 ts_tablet_manager.cc:1428] T c1a5104b5d1245b89d6e3a98e505ddb0 P ed1e5938558345edaccaa79fd9ed26b9: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260926 03:44:46.363685 8877 raft_consensus.cc:348] T c1a5104b5d1245b89d6e3a98e505ddb0 P 8b25e3df61ce40b1818bdb428b3d47a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b25e3df61ce40b1818bdb428b3d47a1" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 45461 } } peers { permanent_uuid: "ed1e5938558345edaccaa79fd9ed26b9" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 40371 } } peers { permanent_uuid: "ae87bdf37fd34fd5b5dc9416b1878954" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 34427 } }
I20260926 03:44:46.363976 8877 raft_consensus.cc:374] T c1a5104b5d1245b89d6e3a98e505ddb0 P 8b25e3df61ce40b1818bdb428b3d47a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:44:46.364075 8877 raft_consensus.cc:729] T c1a5104b5d1245b89d6e3a98e505ddb0 P 8b25e3df61ce40b1818bdb428b3d47a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b25e3df61ce40b1818bdb428b3d47a1, State: Initialized, Role: FOLLOWER
I20260926 03:44:46.364363 8877 consensus_queue.cc:260] T c1a5104b5d1245b89d6e3a98e505ddb0 P 8b25e3df61ce40b1818bdb428b3d47a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b25e3df61ce40b1818bdb428b3d47a1" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 45461 } } peers { permanent_uuid: "ed1e5938558345edaccaa79fd9ed26b9" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 40371 } } peers { permanent_uuid: "ae87bdf37fd34fd5b5dc9416b1878954" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 34427 } }
I20260926 03:44:46.364954 8877 ts_tablet_manager.cc:1428] T c1a5104b5d1245b89d6e3a98e505ddb0 P 8b25e3df61ce40b1818bdb428b3d47a1: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260926 03:44:46.371199 8879 tablet_bootstrap.cc:492] T c1a5104b5d1245b89d6e3a98e505ddb0 P ae87bdf37fd34fd5b5dc9416b1878954: No bootstrap required, opened a new log
I20260926 03:44:46.371384 8879 ts_tablet_manager.cc:1397] T c1a5104b5d1245b89d6e3a98e505ddb0 P ae87bdf37fd34fd5b5dc9416b1878954: Time spent bootstrapping tablet: real 0.019s user 0.001s sys 0.006s
I20260926 03:44:46.372212 8879 raft_consensus.cc:348] T c1a5104b5d1245b89d6e3a98e505ddb0 P ae87bdf37fd34fd5b5dc9416b1878954 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b25e3df61ce40b1818bdb428b3d47a1" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 45461 } } peers { permanent_uuid: "ed1e5938558345edaccaa79fd9ed26b9" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 40371 } } peers { permanent_uuid: "ae87bdf37fd34fd5b5dc9416b1878954" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 34427 } }
I20260926 03:44:46.372526 8879 raft_consensus.cc:374] T c1a5104b5d1245b89d6e3a98e505ddb0 P ae87bdf37fd34fd5b5dc9416b1878954 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:44:46.372607 8879 raft_consensus.cc:729] T c1a5104b5d1245b89d6e3a98e505ddb0 P ae87bdf37fd34fd5b5dc9416b1878954 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae87bdf37fd34fd5b5dc9416b1878954, State: Initialized, Role: FOLLOWER
I20260926 03:44:46.373101 8879 consensus_queue.cc:260] T c1a5104b5d1245b89d6e3a98e505ddb0 P ae87bdf37fd34fd5b5dc9416b1878954 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b25e3df61ce40b1818bdb428b3d47a1" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 45461 } } peers { permanent_uuid: "ed1e5938558345edaccaa79fd9ed26b9" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 40371 } } peers { permanent_uuid: "ae87bdf37fd34fd5b5dc9416b1878954" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 34427 } }
I20260926 03:44:46.373761 8879 ts_tablet_manager.cc:1428] T c1a5104b5d1245b89d6e3a98e505ddb0 P ae87bdf37fd34fd5b5dc9416b1878954: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 03:44:46.675042 8885 raft_consensus.cc:482] T c1a5104b5d1245b89d6e3a98e505ddb0 P ae87bdf37fd34fd5b5dc9416b1878954 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 03:44:46.675309 8885 raft_consensus.cc:504] T c1a5104b5d1245b89d6e3a98e505ddb0 P ae87bdf37fd34fd5b5dc9416b1878954 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b25e3df61ce40b1818bdb428b3d47a1" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 45461 } } peers { permanent_uuid: "ed1e5938558345edaccaa79fd9ed26b9" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 40371 } } peers { permanent_uuid: "ae87bdf37fd34fd5b5dc9416b1878954" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 34427 } }
I20260926 03:44:46.676772 8885 leader_election.cc:288] T c1a5104b5d1245b89d6e3a98e505ddb0 P ae87bdf37fd34fd5b5dc9416b1878954 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8b25e3df61ce40b1818bdb428b3d47a1 (127.0.194.195:45461), ed1e5938558345edaccaa79fd9ed26b9 (127.0.194.193:40371)
I20260926 03:44:46.683672 8876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:46.677230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.194:36429 (local address 127.0.194.193:40371)
0926 03:44:46.677463 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0926 03:44:46.677471 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:44:46.678463 (+ 992us) server_negotiation.cc:407] Connection header received
0926 03:44:46.678648 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:44:46.678651 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:44:46.678706 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:44:46.678826 (+ 120us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:44:46.679470 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:46.680135 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:46.681950 (+ 1815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:46.682721 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:46.682901 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:46.682972 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 03:44:46.683265 (+ 293us) server_negotiation.cc:299] Negotiation successful
0926 03:44:46.683389 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260926 03:44:46.683758 8964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:46.677024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.193:40371 (local address 127.0.194.194:36429)
0926 03:44:46.678354 (+ 1330us) negotiation.cc:107] Waiting for socket to connect
0926 03:44:46.678368 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 03:44:46.678503 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:44:46.678945 (+ 442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:44:46.678951 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:44:46.678970 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:44:46.679325 (+ 355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:46.679332 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:46.680266 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:46.680273 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:46.681803 (+ 1530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:46.681814 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:46.682916 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:46.682922 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:46.683108 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:46.683136 (+ 28us) client_negotiation.cc:769] Sending connection context
0926 03:44:46.683241 (+ 105us) client_negotiation.cc:240] Negotiation successful
0926 03:44:46.683365 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1225,"mutex_wait_us":41,"thread_start_us":1087,"threads_started":1}
I20260926 03:44:46.683910 8963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:46.676969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.195:45461 (local address 127.0.194.194:48073)
0926 03:44:46.678208 (+ 1239us) negotiation.cc:107] Waiting for socket to connect
0926 03:44:46.678228 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 03:44:46.678399 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:44:46.678752 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:44:46.678758 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:44:46.678793 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:44:46.679051 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:46.679060 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:46.680113 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:46.680118 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:46.681810 (+ 1692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:46.681822 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:46.683044 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:46.683050 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:46.683228 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:46.683250 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 03:44:46.683355 (+ 105us) client_negotiation.cc:240] Negotiation successful
0926 03:44:46.683475 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1114,"mutex_wait_us":3,"thread_start_us":960,"threads_started":1}
I20260926 03:44:46.684185 8965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:46.677041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.194:48073 (local address 127.0.194.195:45461)
0926 03:44:46.677928 (+ 887us) server_negotiation.cc:206] Beginning negotiation
0926 03:44:46.677936 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:44:46.678327 (+ 391us) server_negotiation.cc:407] Connection header received
0926 03:44:46.678545 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:44:46.678549 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:44:46.678604 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:44:46.678706 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:44:46.679240 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:46.679971 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:46.681957 (+ 1986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:46.682821 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:46.683018 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:46.683093 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 03:44:46.683379 (+ 286us) server_negotiation.cc:299] Negotiation successful
0926 03:44:46.683511 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":733,"thread_start_us":602,"threads_started":1}
I20260926 03:44:46.684747 8524 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1a5104b5d1245b89d6e3a98e505ddb0" candidate_uuid: "ae87bdf37fd34fd5b5dc9416b1878954" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed1e5938558345edaccaa79fd9ed26b9" is_pre_election: true
I20260926 03:44:46.684842 8807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1a5104b5d1245b89d6e3a98e505ddb0" candidate_uuid: "ae87bdf37fd34fd5b5dc9416b1878954" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b25e3df61ce40b1818bdb428b3d47a1" is_pre_election: true
I20260926 03:44:46.684971 8524 raft_consensus.cc:2454] T c1a5104b5d1245b89d6e3a98e505ddb0 P ed1e5938558345edaccaa79fd9ed26b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae87bdf37fd34fd5b5dc9416b1878954 in term 0.
I20260926 03:44:46.685096 8807 raft_consensus.cc:2454] T c1a5104b5d1245b89d6e3a98e505ddb0 P 8b25e3df61ce40b1818bdb428b3d47a1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae87bdf37fd34fd5b5dc9416b1878954 in term 0.
I20260926 03:44:46.685496 8597 leader_election.cc:302] T c1a5104b5d1245b89d6e3a98e505ddb0 P ae87bdf37fd34fd5b5dc9416b1878954 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 8b25e3df61ce40b1818bdb428b3d47a1, ae87bdf37fd34fd5b5dc9416b1878954, ed1e5938558345edaccaa79fd9ed26b9; no voters:
I20260926 03:44:46.685895 8885 raft_consensus.cc:2781] T c1a5104b5d1245b89d6e3a98e505ddb0 P ae87bdf37fd34fd5b5dc9416b1878954 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 03:44:46.686017 8885 raft_consensus.cc:482] T c1a5104b5d1245b89d6e3a98e505ddb0 P ae87bdf37fd34fd5b5dc9416b1878954 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 03:44:46.686115 8885 raft_consensus.cc:3037] T c1a5104b5d1245b89d6e3a98e505ddb0 P ae87bdf37fd34fd5b5dc9416b1878954 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:44:46.688311 8885 raft_consensus.cc:504] T c1a5104b5d1245b89d6e3a98e505ddb0 P ae87bdf37fd34fd5b5dc9416b1878954 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b25e3df61ce40b1818bdb428b3d47a1" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 45461 } } peers { permanent_uuid: "ed1e5938558345edaccaa79fd9ed26b9" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 40371 } } peers { permanent_uuid: "ae87bdf37fd34fd5b5dc9416b1878954" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 34427 } }
I20260926 03:44:46.688839 8885 leader_election.cc:288] T c1a5104b5d1245b89d6e3a98e505ddb0 P ae87bdf37fd34fd5b5dc9416b1878954 [CANDIDATE]: Term 1 election: Requested vote from peers 8b25e3df61ce40b1818bdb428b3d47a1 (127.0.194.195:45461), ed1e5938558345edaccaa79fd9ed26b9 (127.0.194.193:40371)
I20260926 03:44:46.689285 8524 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1a5104b5d1245b89d6e3a98e505ddb0" candidate_uuid: "ae87bdf37fd34fd5b5dc9416b1878954" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed1e5938558345edaccaa79fd9ed26b9"
I20260926 03:44:46.689301 8807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1a5104b5d1245b89d6e3a98e505ddb0" candidate_uuid: "ae87bdf37fd34fd5b5dc9416b1878954" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b25e3df61ce40b1818bdb428b3d47a1"
I20260926 03:44:46.689469 8524 raft_consensus.cc:3037] T c1a5104b5d1245b89d6e3a98e505ddb0 P ed1e5938558345edaccaa79fd9ed26b9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:44:46.689492 8807 raft_consensus.cc:3037] T c1a5104b5d1245b89d6e3a98e505ddb0 P 8b25e3df61ce40b1818bdb428b3d47a1 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:44:46.691437 8524 raft_consensus.cc:2454] T c1a5104b5d1245b89d6e3a98e505ddb0 P ed1e5938558345edaccaa79fd9ed26b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae87bdf37fd34fd5b5dc9416b1878954 in term 1.
I20260926 03:44:46.691565 8807 raft_consensus.cc:2454] T c1a5104b5d1245b89d6e3a98e505ddb0 P 8b25e3df61ce40b1818bdb428b3d47a1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae87bdf37fd34fd5b5dc9416b1878954 in term 1.
I20260926 03:44:46.691833 8597 leader_election.cc:302] T c1a5104b5d1245b89d6e3a98e505ddb0 P ae87bdf37fd34fd5b5dc9416b1878954 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae87bdf37fd34fd5b5dc9416b1878954, ed1e5938558345edaccaa79fd9ed26b9; no voters:
I20260926 03:44:46.692122 8885 raft_consensus.cc:2781] T c1a5104b5d1245b89d6e3a98e505ddb0 P ae87bdf37fd34fd5b5dc9416b1878954 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:44:46.692492 8885 raft_consensus.cc:686] T c1a5104b5d1245b89d6e3a98e505ddb0 P ae87bdf37fd34fd5b5dc9416b1878954 [term 1 LEADER]: Becoming Leader. State: Replica: ae87bdf37fd34fd5b5dc9416b1878954, State: Running, Role: LEADER
I20260926 03:44:46.692852 8885 consensus_queue.cc:237] T c1a5104b5d1245b89d6e3a98e505ddb0 P ae87bdf37fd34fd5b5dc9416b1878954 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b25e3df61ce40b1818bdb428b3d47a1" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 45461 } } peers { permanent_uuid: "ed1e5938558345edaccaa79fd9ed26b9" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 40371 } } peers { permanent_uuid: "ae87bdf37fd34fd5b5dc9416b1878954" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 34427 } }
I20260926 03:44:46.696259 8367 catalog_manager.cc:5495] T c1a5104b5d1245b89d6e3a98e505ddb0 P ae87bdf37fd34fd5b5dc9416b1878954 reported cstate change: term changed from 0 to 1, leader changed from <none> to ae87bdf37fd34fd5b5dc9416b1878954 (127.0.194.194). New cstate: current_term: 1 leader_uuid: "ae87bdf37fd34fd5b5dc9416b1878954" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b25e3df61ce40b1818bdb428b3d47a1" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 45461 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ed1e5938558345edaccaa79fd9ed26b9" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 40371 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae87bdf37fd34fd5b5dc9416b1878954" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 34427 } health_report { overall_health: HEALTHY } } }
I20260926 03:44:46.744200 8969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:46.738782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45644 (local address 127.0.194.194:34427)
0926 03:44:46.739318 (+ 536us) server_negotiation.cc:206] Beginning negotiation
0926 03:44:46.739322 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:44:46.739340 (+ 18us) server_negotiation.cc:407] Connection header received
0926 03:44:46.739419 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:44:46.739423 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:44:46.739489 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:44:46.739625 (+ 136us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:44:46.739959 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:46.740743 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:46.742023 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:46.742413 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:46.742501 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:46.742961 (+ 460us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:44:46.743560 (+ 599us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:44:46.743673 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0926 03:44:46.743771 (+ 98us) server_negotiation.cc:299] Negotiation successful
0926 03:44:46.743832 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":438,"thread_start_us":316,"threads_started":1}
I20260926 03:44:46.758718 8876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:46.753890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.194:36533 (local address 127.0.194.193:40371)
0926 03:44:46.754049 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0926 03:44:46.754053 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:44:46.754331 (+ 278us) server_negotiation.cc:407] Connection header received
0926 03:44:46.754559 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:44:46.754564 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:44:46.754632 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:44:46.754714 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:44:46.755220 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:46.755931 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:46.757239 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:46.757647 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:46.757740 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:46.758167 (+ 427us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:44:46.758261 (+ 94us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:44:46.758331 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 03:44:46.758446 (+ 115us) server_negotiation.cc:299] Negotiation successful
0926 03:44:46.758529 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 03:44:46.758905 8972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:46.753651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.193:40371 (local address 127.0.194.194:36533)
0926 03:44:46.754243 (+ 592us) negotiation.cc:107] Waiting for socket to connect
0926 03:44:46.754259 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 03:44:46.754371 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:44:46.754798 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:44:46.754803 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:44:46.754820 (+ 17us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 03:44:46.755072 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:46.755080 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:46.756050 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:46.756054 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:46.757067 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:46.757077 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:46.757801 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:46.757805 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:46.757922 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:46.757969 (+ 47us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 03:44:46.758345 (+ 376us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 03:44:46.758352 (+ 7us) client_negotiation.cc:769] Sending connection context
0926 03:44:46.758409 (+ 57us) client_negotiation.cc:240] Negotiation successful
0926 03:44:46.758500 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":297,"threads_started":1}
I20260926 03:44:46.766515 8807 raft_consensus.cc:1261] T c1a5104b5d1245b89d6e3a98e505ddb0 P 8b25e3df61ce40b1818bdb428b3d47a1 [term 1 FOLLOWER]: Refusing update from remote peer ae87bdf37fd34fd5b5dc9416b1878954: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:44:46.766789 8524 raft_consensus.cc:1261] T c1a5104b5d1245b89d6e3a98e505ddb0 P ed1e5938558345edaccaa79fd9ed26b9 [term 1 FOLLOWER]: Refusing update from remote peer ae87bdf37fd34fd5b5dc9416b1878954: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:44:46.767146 8885 consensus_queue.cc:1035] T c1a5104b5d1245b89d6e3a98e505ddb0 P ae87bdf37fd34fd5b5dc9416b1878954 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b25e3df61ce40b1818bdb428b3d47a1" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 45461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:44:46.767499 8966 consensus_queue.cc:1035] T c1a5104b5d1245b89d6e3a98e505ddb0 P ae87bdf37fd34fd5b5dc9416b1878954 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed1e5938558345edaccaa79fd9ed26b9" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 40371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:44:46.794324 8969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:46.789013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.193:58203 (local address 127.0.194.194:34427)
0926 03:44:46.789180 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0926 03:44:46.789185 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:44:46.789622 (+ 437us) server_negotiation.cc:407] Connection header received
0926 03:44:46.789767 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:44:46.789772 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:44:46.789848 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:44:46.789921 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:44:46.790328 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:46.791194 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:46.792550 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:46.793195 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:46.793292 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:46.793611 (+ 319us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:44:46.793701 (+ 90us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:44:46.793785 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 03:44:46.794060 (+ 275us) server_negotiation.cc:299] Negotiation successful
0926 03:44:46.794122 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 03:44:46.794418 8978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:46.789029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.194:34427 (local address 127.0.194.193:58203)
0926 03:44:46.789544 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0926 03:44:46.789559 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 03:44:46.789659 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:44:46.789988 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:44:46.789991 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:44:46.790001 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 03:44:46.790207 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:46.790216 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:46.791337 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:46.791342 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:46.792417 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:46.792429 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:46.793327 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:46.793331 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:46.793445 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:46.793482 (+ 37us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 03:44:46.793848 (+ 366us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 03:44:46.793859 (+ 11us) client_negotiation.cc:769] Sending connection context
0926 03:44:46.794026 (+ 167us) client_negotiation.cc:240] Negotiation successful
0926 03:44:46.794147 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":277,"threads_started":1}
I20260926 03:44:46.828922 8367 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56360:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:44:46.829200 8367 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56360:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:44:46.832681 8367 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f8f4c2ad1ed14a06a9cac074539f400f: Sending DeleteTablet for 3 replicas of tablet c1a5104b5d1245b89d6e3a98e505ddb0
I20260926 03:44:46.833551 8786 tablet_service.cc:1514] Processing DeleteTablet for tablet c1a5104b5d1245b89d6e3a98e505ddb0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:44:46 UTC) from {username='oryx', principal='oryx/127.0.194.254@KRBTEST.COM'} at 127.0.0.1:47720
I20260926 03:44:46.833551 8504 tablet_service.cc:1514] Processing DeleteTablet for tablet c1a5104b5d1245b89d6e3a98e505ddb0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:44:46 UTC) from {username='oryx', principal='oryx/127.0.194.254@KRBTEST.COM'} at 127.0.0.1:42844
I20260926 03:44:46.834018 8642 tablet_service.cc:1514] Processing DeleteTablet for tablet c1a5104b5d1245b89d6e3a98e505ddb0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:44:46 UTC) from {username='oryx', principal='oryx/127.0.194.254@KRBTEST.COM'} at 127.0.0.1:45630
I20260926 03:44:46.835191 779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu pid 8438
I20260926 03:44:46.842918 8983 tablet_replica.cc:330] T c1a5104b5d1245b89d6e3a98e505ddb0 P ed1e5938558345edaccaa79fd9ed26b9: stopping tablet replica
I20260926 03:44:46.843169 8983 raft_consensus.cc:2229] T c1a5104b5d1245b89d6e3a98e505ddb0 P ed1e5938558345edaccaa79fd9ed26b9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:44:46.843426 8983 raft_consensus.cc:2258] T c1a5104b5d1245b89d6e3a98e505ddb0 P ed1e5938558345edaccaa79fd9ed26b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:44:46.844179 8984 tablet_replica.cc:330] T c1a5104b5d1245b89d6e3a98e505ddb0 P ae87bdf37fd34fd5b5dc9416b1878954: stopping tablet replica
I20260926 03:44:46.844404 8984 raft_consensus.cc:2229] T c1a5104b5d1245b89d6e3a98e505ddb0 P ae87bdf37fd34fd5b5dc9416b1878954 [term 1 LEADER]: Raft consensus shutting down.
I20260926 03:44:46.845109 8984 raft_consensus.cc:2258] T c1a5104b5d1245b89d6e3a98e505ddb0 P ae87bdf37fd34fd5b5dc9416b1878954 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:44:46.845646 8982 tablet_replica.cc:330] T c1a5104b5d1245b89d6e3a98e505ddb0 P 8b25e3df61ce40b1818bdb428b3d47a1: stopping tablet replica
I20260926 03:44:46.845911 8982 raft_consensus.cc:2229] T c1a5104b5d1245b89d6e3a98e505ddb0 P 8b25e3df61ce40b1818bdb428b3d47a1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:44:46.846149 8982 raft_consensus.cc:2258] T c1a5104b5d1245b89d6e3a98e505ddb0 P 8b25e3df61ce40b1818bdb428b3d47a1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:44:46.846534 8983 ts_tablet_manager.cc:1905] T c1a5104b5d1245b89d6e3a98e505ddb0 P ed1e5938558345edaccaa79fd9ed26b9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:44:46.847471 8984 ts_tablet_manager.cc:1905] T c1a5104b5d1245b89d6e3a98e505ddb0 P ae87bdf37fd34fd5b5dc9416b1878954: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:44:46.850826 8982 ts_tablet_manager.cc:1905] T c1a5104b5d1245b89d6e3a98e505ddb0 P 8b25e3df61ce40b1818bdb428b3d47a1: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 26 03:44:46 dist-test-slave-n2th krb5kdc[8322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394285, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.194.193@KRBTEST.COM
I20260926 03:44:46.853899 8983 ts_tablet_manager.cc:1918] T c1a5104b5d1245b89d6e3a98e505ddb0 P ed1e5938558345edaccaa79fd9ed26b9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 03:44:46.854039 8983 log.cc:1196] T c1a5104b5d1245b89d6e3a98e505ddb0 P ed1e5938558345edaccaa79fd9ed26b9: Deleting WAL directory at /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790394221564982-779-0/minicluster-data/ts-0/wal/wals/c1a5104b5d1245b89d6e3a98e505ddb0
I20260926 03:44:46.854539 8983 ts_tablet_manager.cc:1939] T c1a5104b5d1245b89d6e3a98e505ddb0 P ed1e5938558345edaccaa79fd9ed26b9: Deleting consensus metadata
I20260926 03:44:46.855803 8354 catalog_manager.cc:4841] TS ed1e5938558345edaccaa79fd9ed26b9 (127.0.194.193:40371): tablet c1a5104b5d1245b89d6e3a98e505ddb0 (table test-table [id=ae36445423c64444bb1c74a38ebbb310]) successfully deleted
I20260926 03:44:46.857676 8982 ts_tablet_manager.cc:1918] T c1a5104b5d1245b89d6e3a98e505ddb0 P 8b25e3df61ce40b1818bdb428b3d47a1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 03:44:46.857831 8982 log.cc:1196] T c1a5104b5d1245b89d6e3a98e505ddb0 P 8b25e3df61ce40b1818bdb428b3d47a1: Deleting WAL directory at /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790394221564982-779-0/minicluster-data/ts-2/wal/wals/c1a5104b5d1245b89d6e3a98e505ddb0
I20260926 03:44:46.858261 8982 ts_tablet_manager.cc:1939] T c1a5104b5d1245b89d6e3a98e505ddb0 P 8b25e3df61ce40b1818bdb428b3d47a1: Deleting consensus metadata
I20260926 03:44:46.859683 8351 catalog_manager.cc:4841] TS 8b25e3df61ce40b1818bdb428b3d47a1 (127.0.194.195:45461): tablet c1a5104b5d1245b89d6e3a98e505ddb0 (table test-table [id=ae36445423c64444bb1c74a38ebbb310]) successfully deleted
I20260926 03:44:46.862093 8876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:46.836085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42892 (local address 127.0.194.193:40371)
0926 03:44:46.836372 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0926 03:44:46.836382 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 03:44:46.842959 (+ 6577us) server_negotiation.cc:407] Connection header received
0926 03:44:46.843268 (+ 309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:44:46.843277 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:44:46.843358 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:44:46.843454 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:44:46.845639 (+ 2185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:46.846665 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:46.848606 (+ 1941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:46.849342 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:46.849436 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:46.855262 (+ 5826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:44:46.855282 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:44:46.855302 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:44:46.855341 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:44:46.858988 (+ 3647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:46.859902 (+ 914us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:46.859914 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:46.859917 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:46.860078 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:46.860449 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:46.860455 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:46.860458 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:46.860821 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:44:46.861157 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0926 03:44:46.861439 (+ 282us) server_negotiation.cc:299] Negotiation successful
0926 03:44:46.861729 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260926 03:44:46.862777 8569 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:44:46.863751 8984 ts_tablet_manager.cc:1918] T c1a5104b5d1245b89d6e3a98e505ddb0 P ae87bdf37fd34fd5b5dc9416b1878954: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 03:44:46.864001 8984 log.cc:1196] T c1a5104b5d1245b89d6e3a98e505ddb0 P ae87bdf37fd34fd5b5dc9416b1878954: Deleting WAL directory at /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790394221564982-779-0/minicluster-data/ts-1/wal/wals/c1a5104b5d1245b89d6e3a98e505ddb0
I20260926 03:44:46.864655 8984 ts_tablet_manager.cc:1939] T c1a5104b5d1245b89d6e3a98e505ddb0 P ae87bdf37fd34fd5b5dc9416b1878954: Deleting consensus metadata
I20260926 03:44:46.866535 8353 catalog_manager.cc:4841] TS ae87bdf37fd34fd5b5dc9416b1878954 (127.0.194.194:34427): tablet c1a5104b5d1245b89d6e3a98e505ddb0 (table test-table [id=ae36445423c64444bb1c74a38ebbb310]) successfully deleted
I20260926 03:44:47.010900 779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu with pid 8438
I20260926 03:44:47.046871 779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu pid 8578
Sep 26 03:44:47 dist-test-slave-n2th krb5kdc[8322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394285, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.194.194@KRBTEST.COM
I20260926 03:44:47.062551 8969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:47.047621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45652 (local address 127.0.194.194:34427)
0926 03:44:47.047823 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0926 03:44:47.047830 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:44:47.047848 (+ 18us) server_negotiation.cc:407] Connection header received
0926 03:44:47.048070 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:44:47.048074 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:44:47.048159 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:44:47.048279 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:44:47.049477 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:47.050228 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:47.051544 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:47.052000 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:47.052135 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:47.056324 (+ 4189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:44:47.056336 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:44:47.056350 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:44:47.056389 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:44:47.059640 (+ 3251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:47.060455 (+ 815us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:47.060463 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:47.060466 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:47.060604 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:47.060993 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:47.060999 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:47.061002 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:47.061353 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:44:47.061665 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0926 03:44:47.061979 (+ 314us) server_negotiation.cc:299] Negotiation successful
0926 03:44:47.062199 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 03:44:47.063288 8707 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:44:47.188648 779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu with pid 8578
I20260926 03:44:47.208654 779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu pid 8717
Sep 26 03:44:47 dist-test-slave-n2th krb5kdc[8322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394285, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.194.195@KRBTEST.COM
I20260926 03:44:47.223618 8988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:47.209410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47728 (local address 127.0.194.195:45461)
0926 03:44:47.209913 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0926 03:44:47.209917 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:44:47.209934 (+ 17us) server_negotiation.cc:407] Connection header received
0926 03:44:47.209988 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:44:47.209992 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:44:47.210050 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:44:47.210142 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:44:47.211421 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:47.212137 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:47.213298 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:47.213683 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:47.213780 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:47.218055 (+ 4275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:44:47.218071 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:44:47.218087 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:44:47.218119 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:44:47.220770 (+ 2651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:47.221455 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:47.221463 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:47.221465 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:47.221591 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:47.221955 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:47.221962 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:47.221965 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:47.222311 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:44:47.222631 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0926 03:44:47.223055 (+ 424us) server_negotiation.cc:299] Negotiation successful
0926 03:44:47.223286 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":402,"thread_start_us":274,"threads_started":1}
I20260926 03:44:47.224270 8847 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:44:47.345798 779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu with pid 8717
I20260926 03:44:47.367568 779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu pid 8335
I20260926 03:44:47.368708 8407 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:44:47.482311 779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu with pid 8335
2026-09-26T03:44:47Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (7002 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790394221564982-779-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:44:47 dist-test-slave-n2th krb5kdc[8993](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:44:47 dist-test-slave-n2th krb5kdc[8993](info): set up 2 sockets
Sep 26 03:44:47 dist-test-slave-n2th krb5kdc[8993](info): commencing operation
krb5kdc: starting...
W20260926 03:44:48.579727 779 mini_kdc.cc:121] Time spent starting KDC: real 1.053s user 0.000s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790394221564982-779-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790394221564982-779-0/krb5kdc/test-user.keytab.
Sep 26 03:44:48 dist-test-slave-n2th krb5kdc[8993](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394288, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:44:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:44:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.194.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.194.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.194.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790394221564982-779-0/krb5kdc/kudu_127.0.194.254.keytab.
Entry for principal kudu/127.0.194.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790394221564982-779-0/krb5kdc/kudu_127.0.194.254.keytab.
I20260926 03:44:48.798504 779 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
/tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790394221564982-779-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790394221564982-779-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790394221564982-779-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790394221564982-779-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.194.254:43279
--webserver_interface=127.0.194.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790394221564982-779-0/krb5kdc/kudu_127.0.194.254.keytab
--principal=kudu/127.0.194.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.194.212:46631
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.194.254:43279
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790394221564982-779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790394221564982-779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790394221564982-779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:44:49.124599 9006 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:44:49.124964 9006 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:44:49.125123 9006 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:44:49.133095 9006 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:44:49.133256 9006 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:44:49.133359 9006 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:44:49.133445 9006 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:44:49.133526 9006 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:44:49.143541 9006 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.194.212:46631
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790394221564982-779-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790394221564982-779-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.194.254:43279
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790394221564982-779-0/krb5kdc/kudu_127.0.194.254.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.194.254:43279
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790394221564982-779-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.194.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790394221564982-779-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:44:49.144886 9006 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:44:49.146210 9006 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:44:49.158293 9014 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:44:49.158450 9012 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:44:49.158754 9006 server_base.cc:1034] running on GCE node
W20260926 03:44:49.159020 9011 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:44:49.159902 9006 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:44:49.161265 9006 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:44:49.162497 9006 hybrid_clock.cc:648] HybridClock initialized: now 1790394289162465 us; error 47 us; skew 500 ppm
I20260926 03:44:49.162896 9006 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-26T03:44:49Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1743 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790394221564982-779-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:44:49 dist-test-slave-n2th krb5kdc[9020](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:44:49 dist-test-slave-n2th krb5kdc[9020](info): set up 2 sockets
Sep 26 03:44:49 dist-test-slave-n2th krb5kdc[9020](info): commencing operation
krb5kdc: starting...
W20260926 03:44:50.323705 779 mini_kdc.cc:121] Time spent starting KDC: real 1.054s user 0.002s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790394221564982-779-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790394221564982-779-0/krb5kdc/test-user.keytab.
Sep 26 03:44:50 dist-test-slave-n2th krb5kdc[9020](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394290, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:44:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:44:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.194.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.194.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.194.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790394221564982-779-0/krb5kdc/kudu_127.0.194.254.keytab.
Entry for principal kudu/127.0.194.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790394221564982-779-0/krb5kdc/kudu_127.0.194.254.keytab.
I20260926 03:44:50.545697 779 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
/tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790394221564982-779-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790394221564982-779-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790394221564982-779-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790394221564982-779-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.194.254:36233
--webserver_interface=127.0.194.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790394221564982-779-0/krb5kdc/kudu_127.0.194.254.keytab
--principal=kudu/127.0.194.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.194.212:46875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.194.254:36233
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790394221564982-779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790394221564982-779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790394221564982-779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:44:50.870165 9033 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:44:50.870522 9033 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:44:50.870675 9033 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:44:50.878077 9033 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:44:50.878230 9033 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:44:50.878340 9033 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:44:50.878443 9033 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:44:50.878528 9033 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:44:50.888695 9033 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.194.212:46875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790394221564982-779-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790394221564982-779-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.194.254:36233
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790394221564982-779-0/krb5kdc/kudu_127.0.194.254.keytab
--principal=kudu/127.0.194.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.194.254:36233
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790394221564982-779-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.194.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790394221564982-779-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:44:50.890398 9033 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:44:50.892135 9033 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:44:50.904124 9038 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:44:50.904160 9041 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:44:50.904141 9039 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:44:50.905344 9033 server_base.cc:1034] running on GCE node
I20260926 03:44:50.906133 9033 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:44:50.907873 9033 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:44:50.909147 9033 hybrid_clock.cc:648] HybridClock initialized: now 1790394290909058 us; error 86 us; skew 500 ppm
I20260926 03:44:50.909618 9033 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:44:50 dist-test-slave-n2th krb5kdc[9020](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394290, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:44:50.919798 9033 init.cc:377] Logged in from keytab as kudu/127.0.194.254@KRBTEST.COM (short username kudu)
I20260926 03:44:50.921406 9033 webserver.cc:460] Webserver started at http://127.0.194.254:41557/ using document root <none> and password file <none>
I20260926 03:44:50.922128 9033 fs_manager.cc:359] Metadata directory not provided
I20260926 03:44:50.922281 9033 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:44:50.922591 9033 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:44:50.925580 9033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790394221564982-779-0/minicluster-data/master-0/data/instance:
uuid: "47aeac76c7fd4f84a64b43a172dfd695"
format_stamp: "Formatted at 2026-09-26 03:44:50 on dist-test-slave-n2th"
server_key: "dd019ebb6937c0c170a1140a9de40309"
server_key_iv: "1e4fc0e831863a411ac22fcf8b5c5d05"
server_key_version: "encryptionkey@0"
I20260926 03:44:50.926357 9033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790394221564982-779-0/minicluster-data/master-0/wal/instance:
uuid: "47aeac76c7fd4f84a64b43a172dfd695"
format_stamp: "Formatted at 2026-09-26 03:44:50 on dist-test-slave-n2th"
server_key: "dd019ebb6937c0c170a1140a9de40309"
server_key_iv: "1e4fc0e831863a411ac22fcf8b5c5d05"
server_key_version: "encryptionkey@0"
I20260926 03:44:50.931394 9033 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260926 03:44:50.934659 9048 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:44:50.935981 9033 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 03:44:50.936160 9033 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790394221564982-779-0/minicluster-data/master-0/data,/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790394221564982-779-0/minicluster-data/master-0/wal
uuid: "47aeac76c7fd4f84a64b43a172dfd695"
format_stamp: "Formatted at 2026-09-26 03:44:50 on dist-test-slave-n2th"
server_key: "dd019ebb6937c0c170a1140a9de40309"
server_key_iv: "1e4fc0e831863a411ac22fcf8b5c5d05"
server_key_version: "encryptionkey@0"
I20260926 03:44:50.936357 9033 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790394221564982-779-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790394221564982-779-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790394221564982-779-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:44:50.952240 9033 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:44:50.956169 9033 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:44:50.956465 9033 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:44:50.974828 9033 rpc_server.cc:307] RPC server started. Bound to: 127.0.194.254:36233
I20260926 03:44:50.974833 9099 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.194.254:36233 every 8 connection(s)
I20260926 03:44:50.976352 9033 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790394221564982-779-0/minicluster-data/master-0/data/info.pb
I20260926 03:44:50.976840 779 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu as pid 9033
I20260926 03:44:50.977049 779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790394221564982-779-0/minicluster-data/master-0/wal/instance
I20260926 03:44:50.977869 779 external_mini_cluster.cc:1428] Setting key f72bb491431deaeb5a8b3e20b7ce2923
I20260926 03:44:50.980940 9100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 26 03:44:50 dist-test-slave-n2th krb5kdc[9020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394290, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.194.254@KRBTEST.COM
I20260926 03:44:50.993705 9100 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47aeac76c7fd4f84a64b43a172dfd695: Bootstrap starting.
I20260926 03:44:50.997026 9100 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 47aeac76c7fd4f84a64b43a172dfd695: Neither blocks nor log segments found. Creating new log.
I20260926 03:44:50.998292 9100 log.cc:824] T 00000000000000000000000000000000 P 47aeac76c7fd4f84a64b43a172dfd695: Log is configured to *not* fsync() on all Append() calls
I20260926 03:44:51.001224 9100 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47aeac76c7fd4f84a64b43a172dfd695: No bootstrap required, opened a new log
I20260926 03:44:51.001662 9103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:50.979913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59350 (local address 127.0.194.254:36233)
0926 03:44:50.980699 (+ 786us) server_negotiation.cc:206] Beginning negotiation
0926 03:44:50.980714 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 03:44:50.980762 (+ 48us) server_negotiation.cc:407] Connection header received
0926 03:44:50.983071 (+ 2309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:44:50.983098 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:44:50.983593 (+ 495us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:44:50.984052 (+ 459us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:44:50.985474 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:50.987584 (+ 2110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:50.988877 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:50.989912 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:50.990100 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:50.994210 (+ 4110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:44:50.994217 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:44:50.994233 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:44:50.994273 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:44:50.997407 (+ 3134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:50.998253 (+ 846us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:50.998259 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:50.998265 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:50.998464 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:50.998917 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:50.998922 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:50.998927 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:50.999458 (+ 531us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:44:50.999753 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0926 03:44:51.000194 (+ 441us) server_negotiation.cc:299] Negotiation successful
0926 03:44:51.000608 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":562,"thread_start_us":315,"threads_started":1}
I20260926 03:44:51.009909 9100 raft_consensus.cc:348] T 00000000000000000000000000000000 P 47aeac76c7fd4f84a64b43a172dfd695 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47aeac76c7fd4f84a64b43a172dfd695" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 36233 } }
I20260926 03:44:51.010295 9100 raft_consensus.cc:374] T 00000000000000000000000000000000 P 47aeac76c7fd4f84a64b43a172dfd695 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:44:51.010414 9100 raft_consensus.cc:729] T 00000000000000000000000000000000 P 47aeac76c7fd4f84a64b43a172dfd695 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47aeac76c7fd4f84a64b43a172dfd695, State: Initialized, Role: FOLLOWER
I20260926 03:44:51.011060 9100 consensus_queue.cc:260] T 00000000000000000000000000000000 P 47aeac76c7fd4f84a64b43a172dfd695 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47aeac76c7fd4f84a64b43a172dfd695" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 36233 } }
I20260926 03:44:51.011288 9100 raft_consensus.cc:388] T 00000000000000000000000000000000 P 47aeac76c7fd4f84a64b43a172dfd695 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:44:51.011415 9100 raft_consensus.cc:482] T 00000000000000000000000000000000 P 47aeac76c7fd4f84a64b43a172dfd695 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:44:51.011588 9100 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 47aeac76c7fd4f84a64b43a172dfd695 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:44:51.013882 9100 raft_consensus.cc:504] T 00000000000000000000000000000000 P 47aeac76c7fd4f84a64b43a172dfd695 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47aeac76c7fd4f84a64b43a172dfd695" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 36233 } }
I20260926 03:44:51.014501 9100 leader_election.cc:302] T 00000000000000000000000000000000 P 47aeac76c7fd4f84a64b43a172dfd695 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 47aeac76c7fd4f84a64b43a172dfd695; no voters:
I20260926 03:44:51.015146 9100 leader_election.cc:288] T 00000000000000000000000000000000 P 47aeac76c7fd4f84a64b43a172dfd695 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:44:51.015297 9105 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 47aeac76c7fd4f84a64b43a172dfd695 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:44:51.016021 9105 raft_consensus.cc:686] T 00000000000000000000000000000000 P 47aeac76c7fd4f84a64b43a172dfd695 [term 1 LEADER]: Becoming Leader. State: Replica: 47aeac76c7fd4f84a64b43a172dfd695, State: Running, Role: LEADER
I20260926 03:44:51.016547 9105 consensus_queue.cc:237] T 00000000000000000000000000000000 P 47aeac76c7fd4f84a64b43a172dfd695 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47aeac76c7fd4f84a64b43a172dfd695" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 36233 } }
I20260926 03:44:51.017277 9100 sys_catalog.cc:564] T 00000000000000000000000000000000 P 47aeac76c7fd4f84a64b43a172dfd695 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:44:51.019733 9107 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47aeac76c7fd4f84a64b43a172dfd695 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 47aeac76c7fd4f84a64b43a172dfd695. Latest consensus state: current_term: 1 leader_uuid: "47aeac76c7fd4f84a64b43a172dfd695" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47aeac76c7fd4f84a64b43a172dfd695" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 36233 } } }
I20260926 03:44:51.020145 9107 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47aeac76c7fd4f84a64b43a172dfd695 [sys.catalog]: This master's current role is: LEADER
I20260926 03:44:51.019727 9106 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47aeac76c7fd4f84a64b43a172dfd695 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "47aeac76c7fd4f84a64b43a172dfd695" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47aeac76c7fd4f84a64b43a172dfd695" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 36233 } } }
I20260926 03:44:51.020802 9106 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47aeac76c7fd4f84a64b43a172dfd695 [sys.catalog]: This master's current role is: LEADER
I20260926 03:44:51.029584 9112 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:44:51.034283 9112 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:44:51.043210 9112 catalog_manager.cc:1347] Generated new cluster ID: fd2f1ea054f543fc90027c6c8e4a745d
I20260926 03:44:51.043342 9112 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:44:51.056244 9112 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:44:51.058220 9112 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:44:51.067293 9112 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 47aeac76c7fd4f84a64b43a172dfd695: Generated new TSK 0
I20260926 03:44:51.068387 9112 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.194.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.194.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.194.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790394221564982-779-0/krb5kdc/kudu_127.0.194.193.keytab.
Entry for principal kudu/127.0.194.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790394221564982-779-0/krb5kdc/kudu_127.0.194.193.keytab.
I20260926 03:44:51.128536 779 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
/tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790394221564982-779-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790394221564982-779-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790394221564982-779-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790394221564982-779-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.194.193:0
--local_ip_for_outbound_sockets=127.0.194.193
--webserver_interface=127.0.194.193
--webserver_port=0
--tserver_master_addrs=127.0.194.254:36233
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790394221564982-779-0/krb5kdc/kudu_127.0.194.193.keytab
--principal=kudu/127.0.194.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.194.212:46875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790394221564982-779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790394221564982-779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790394221564982-779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:44:51.457350 9126 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:44:51.457643 9126 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:44:51.457805 9126 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:44:51.465304 9126 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:44:51.465421 9126 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:44:51.465620 9126 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.194.193
I20260926 03:44:51.475696 9126 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.194.212:46875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790394221564982-779-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790394221564982-779-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790394221564982-779-0/krb5kdc/kudu_127.0.194.193.keytab
--principal=kudu/127.0.194.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.194.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790394221564982-779-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.194.193
--webserver_port=0
--tserver_master_addrs=127.0.194.254:36233
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.194.193
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790394221564982-779-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:44:51.477090 9126 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:44:51.478404 9126 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:44:51.491560 9131 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:44:51.491637 9134 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:44:51.491601 9132 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:44:51.492516 9126 server_base.cc:1034] running on GCE node
I20260926 03:44:51.493180 9126 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:44:51.494144 9126 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:44:51.495380 9126 hybrid_clock.cc:648] HybridClock initialized: now 1790394291495329 us; error 45 us; skew 500 ppm
I20260926 03:44:51.495711 9126 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:44:51 dist-test-slave-n2th krb5kdc[9020](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394291, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:44:51.499002 9126 init.cc:377] Logged in from keytab as kudu/127.0.194.193@KRBTEST.COM (short username kudu)
I20260926 03:44:51.500555 9126 webserver.cc:460] Webserver started at http://127.0.194.193:43277/ using document root <none> and password file <none>
I20260926 03:44:51.501569 9126 fs_manager.cc:359] Metadata directory not provided
I20260926 03:44:51.501736 9126 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:44:51.502053 9126 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:44:51.505138 9126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790394221564982-779-0/minicluster-data/ts-0/data/instance:
uuid: "4386950be0e24073a32223c354a87dd1"
format_stamp: "Formatted at 2026-09-26 03:44:51 on dist-test-slave-n2th"
server_key: "6e648b7fdce962d3d1b1c5f864ece6e9"
server_key_iv: "9d0b5c2c7e4ce14f8cb57486dea9c76d"
server_key_version: "encryptionkey@0"
I20260926 03:44:51.505968 9126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790394221564982-779-0/minicluster-data/ts-0/wal/instance:
uuid: "4386950be0e24073a32223c354a87dd1"
format_stamp: "Formatted at 2026-09-26 03:44:51 on dist-test-slave-n2th"
server_key: "6e648b7fdce962d3d1b1c5f864ece6e9"
server_key_iv: "9d0b5c2c7e4ce14f8cb57486dea9c76d"
server_key_version: "encryptionkey@0"
I20260926 03:44:51.511322 9126 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260926 03:44:51.514942 9141 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:44:51.516276 9126 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 03:44:51.516466 9126 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790394221564982-779-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790394221564982-779-0/minicluster-data/ts-0/wal
uuid: "4386950be0e24073a32223c354a87dd1"
format_stamp: "Formatted at 2026-09-26 03:44:51 on dist-test-slave-n2th"
server_key: "6e648b7fdce962d3d1b1c5f864ece6e9"
server_key_iv: "9d0b5c2c7e4ce14f8cb57486dea9c76d"
server_key_version: "encryptionkey@0"
I20260926 03:44:51.516680 9126 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790394221564982-779-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790394221564982-779-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790394221564982-779-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:44:51.557570 9126 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:44:51.562265 9126 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:44:51.562592 9126 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:44:51.563616 9126 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:44:51.565037 9126 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:44:51.565110 9126 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:44:51.565243 9126 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:44:51.565311 9126 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:44:51.599224 9126 rpc_server.cc:307] RPC server started. Bound to: 127.0.194.193:41821
I20260926 03:44:51.599267 9253 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.194.193:41821 every 8 connection(s)
I20260926 03:44:51.600472 9126 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790394221564982-779-0/minicluster-data/ts-0/data/info.pb
I20260926 03:44:51.603919 779 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu as pid 9126
I20260926 03:44:51.604125 779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790394221564982-779-0/minicluster-data/ts-0/wal/instance
I20260926 03:44:51.604960 779 external_mini_cluster.cc:1428] Setting key 444ea155f6c348f9fb9befd24ec6ccc3
Sep 26 03:44:51 dist-test-slave-n2th krb5kdc[9020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394291, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.193@KRBTEST.COM for kudu/127.0.194.254@KRBTEST.COM
I20260926 03:44:51.620327 9256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:51.602896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.193:59779 (local address 127.0.194.254:36233)
0926 03:44:51.603309 (+ 413us) server_negotiation.cc:206] Beginning negotiation
0926 03:44:51.603314 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:44:51.604483 (+ 1169us) server_negotiation.cc:407] Connection header received
0926 03:44:51.606599 (+ 2116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:44:51.606604 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:44:51.606692 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:44:51.606834 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:44:51.608256 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:51.608944 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:51.610410 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:51.610846 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:51.610975 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:51.614585 (+ 3610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:44:51.614590 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:44:51.614593 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:44:51.614623 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:44:51.617288 (+ 2665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:51.617978 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:51.617985 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:51.617988 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:51.618116 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:51.618562 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:51.618569 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:51.618572 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:51.618906 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:44:51.619049 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0926 03:44:51.619749 (+ 700us) server_negotiation.cc:299] Negotiation successful
0926 03:44:51.619965 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305,"thread_start_us":209,"threads_started":1}
I20260926 03:44:51.621567 9257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:51.603497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:36233 (local address 127.0.194.193:59779)
0926 03:44:51.604290 (+ 793us) negotiation.cc:107] Waiting for socket to connect
0926 03:44:51.604348 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0926 03:44:51.606341 (+ 1993us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:44:51.607056 (+ 715us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:44:51.607063 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:44:51.607547 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:44:51.608087 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:51.608104 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:51.609079 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:51.609087 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:51.610274 (+ 1187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:51.610285 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:51.611056 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:51.611064 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:51.611296 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:51.611969 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:44:51.611991 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:44:51.614364 (+ 2373us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:44:51.617425 (+ 3061us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:44:51.617433 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:44:51.617449 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:44:51.617857 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:44:51.618256 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:44:51.618261 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:44:51.618265 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:44:51.618450 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:44:51.619043 (+ 593us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:44:51.619051 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:44:51.619379 (+ 328us) client_negotiation.cc:769] Sending connection context
0926 03:44:51.619647 (+ 268us) client_negotiation.cc:240] Negotiation successful
0926 03:44:51.620175 (+ 528us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":306,"threads_started":1}
I20260926 03:44:51.623172 9254 heartbeater.cc:347] Connected to a master server at 127.0.194.254:36233
I20260926 03:44:51.623512 9254 heartbeater.cc:464] Registering TS with master...
I20260926 03:44:51.624418 9254 heartbeater.cc:511] Master 127.0.194.254:36233 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.194.194@KRBTEST.COM; defaulting to no policy
I20260926 03:44:51.626658 9065 ts_manager.cc:194] Registered new tserver with Master: 4386950be0e24073a32223c354a87dd1 (127.0.194.193:41821)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.194.194@KRBTEST.COM" created.
I20260926 03:44:51.628264 9065 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.194.193@KRBTEST.COM'} at 127.0.194.193:59779
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.194.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790394221564982-779-0/krb5kdc/kudu_127.0.194.194.keytab.
Entry for principal kudu/127.0.194.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790394221564982-779-0/krb5kdc/kudu_127.0.194.194.keytab.
I20260926 03:44:51.656054 779 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
/tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790394221564982-779-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790394221564982-779-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790394221564982-779-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790394221564982-779-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.194.194:0
--local_ip_for_outbound_sockets=127.0.194.194
--webserver_interface=127.0.194.194
--webserver_port=0
--tserver_master_addrs=127.0.194.254:36233
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790394221564982-779-0/krb5kdc/kudu_127.0.194.194.keytab
--principal=kudu/127.0.194.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.194.212:46875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790394221564982-779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790394221564982-779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790394221564982-779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:44:51.987740 9260 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:44:51.988020 9260 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:44:51.988186 9260 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:44:51.995478 9260 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:44:51.995630 9260 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:44:51.995831 9260 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.194.194
I20260926 03:44:52.005800 9260 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.194.212:46875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790394221564982-779-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790394221564982-779-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790394221564982-779-0/krb5kdc/kudu_127.0.194.194.keytab
--principal=kudu/127.0.194.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.194.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790394221564982-779-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.194.194
--webserver_port=0
--tserver_master_addrs=127.0.194.254:36233
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.194.194
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790394221564982-779-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:44:52.007148 9260 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:44:52.008472 9260 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:44:52.021867 9265 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:44:52.021867 9268 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:44:52.021889 9266 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:44:52.022771 9260 server_base.cc:1034] running on GCE node
I20260926 03:44:52.023418 9260 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:44:52.024396 9260 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:44:52.025633 9260 hybrid_clock.cc:648] HybridClock initialized: now 1790394292025569 us; error 60 us; skew 500 ppm
I20260926 03:44:52.026027 9260 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:44:52 dist-test-slave-n2th krb5kdc[9020](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394292, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:44:52.029249 9260 init.cc:377] Logged in from keytab as kudu/127.0.194.194@KRBTEST.COM (short username kudu)
I20260926 03:44:52.030857 9260 webserver.cc:460] Webserver started at http://127.0.194.194:46253/ using document root <none> and password file <none>
I20260926 03:44:52.031538 9260 fs_manager.cc:359] Metadata directory not provided
I20260926 03:44:52.031636 9260 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:44:52.031891 9260 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:44:52.034847 9260 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790394221564982-779-0/minicluster-data/ts-1/data/instance:
uuid: "32a1a2dbed28471d96e29ce6cb51b40f"
format_stamp: "Formatted at 2026-09-26 03:44:52 on dist-test-slave-n2th"
server_key: "a11c4b3c3d25e319f706c2c2c920e56a"
server_key_iv: "a5151ec1e2965b5e39100e8c39632042"
server_key_version: "encryptionkey@0"
I20260926 03:44:52.035714 9260 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790394221564982-779-0/minicluster-data/ts-1/wal/instance:
uuid: "32a1a2dbed28471d96e29ce6cb51b40f"
format_stamp: "Formatted at 2026-09-26 03:44:52 on dist-test-slave-n2th"
server_key: "a11c4b3c3d25e319f706c2c2c920e56a"
server_key_iv: "a5151ec1e2965b5e39100e8c39632042"
server_key_version: "encryptionkey@0"
I20260926 03:44:52.041298 9260 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 03:44:52.044938 9275 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:44:52.046325 9260 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260926 03:44:52.046517 9260 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790394221564982-779-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790394221564982-779-0/minicluster-data/ts-1/wal
uuid: "32a1a2dbed28471d96e29ce6cb51b40f"
format_stamp: "Formatted at 2026-09-26 03:44:52 on dist-test-slave-n2th"
server_key: "a11c4b3c3d25e319f706c2c2c920e56a"
server_key_iv: "a5151ec1e2965b5e39100e8c39632042"
server_key_version: "encryptionkey@0"
I20260926 03:44:52.046744 9260 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790394221564982-779-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790394221564982-779-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790394221564982-779-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:44:52.060511 9260 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:44:52.064901 9260 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:44:52.065222 9260 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:44:52.066244 9260 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:44:52.067824 9260 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:44:52.067925 9260 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:44:52.068068 9260 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:44:52.068171 9260 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:44:52.102062 9260 rpc_server.cc:307] RPC server started. Bound to: 127.0.194.194:42665
I20260926 03:44:52.102084 9387 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.194.194:42665 every 8 connection(s)
I20260926 03:44:52.103462 9260 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790394221564982-779-0/minicluster-data/ts-1/data/info.pb
I20260926 03:44:52.107601 779 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu as pid 9260
I20260926 03:44:52.107795 779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790394221564982-779-0/minicluster-data/ts-1/wal/instance
I20260926 03:44:52.108361 779 external_mini_cluster.cc:1428] Setting key 8b366116170fc933dd2ce8e8e30acf40
Sep 26 03:44:52 dist-test-slave-n2th krb5kdc[9020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394292, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.194@KRBTEST.COM for kudu/127.0.194.254@KRBTEST.COM
I20260926 03:44:52.127858 9256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:52.106072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.194:43081 (local address 127.0.194.254:36233)
0926 03:44:52.106269 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0926 03:44:52.106274 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:44:52.107712 (+ 1438us) server_negotiation.cc:407] Connection header received
0926 03:44:52.109889 (+ 2177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:44:52.109895 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:44:52.109970 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:44:52.110086 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:44:52.112236 (+ 2150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:52.112954 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:52.115507 (+ 2553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:52.115945 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:52.116045 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:52.120257 (+ 4212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:44:52.120266 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:44:52.120274 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:44:52.120353 (+ 79us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:44:52.124336 (+ 3983us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:52.125151 (+ 815us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:52.125159 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:52.125166 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:52.125359 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:52.125802 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:52.125809 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:52.125812 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:52.126194 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:44:52.126356 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0926 03:44:52.127130 (+ 774us) server_negotiation.cc:299] Negotiation successful
0926 03:44:52.127407 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 03:44:52.128573 9390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:52.106605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:36233 (local address 127.0.194.194:43081)
0926 03:44:52.107478 (+ 873us) negotiation.cc:107] Waiting for socket to connect
0926 03:44:52.107532 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0926 03:44:52.109592 (+ 2060us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:44:52.110353 (+ 761us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:44:52.110363 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:44:52.111140 (+ 777us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:44:52.112034 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:52.112064 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:52.113122 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:52.113130 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:52.115352 (+ 2222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:52.115370 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:52.116196 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:52.116205 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:52.116547 (+ 342us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:52.117327 (+ 780us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:44:52.117362 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:44:52.119994 (+ 2632us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:44:52.124507 (+ 4513us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:44:52.124515 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:44:52.124529 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:44:52.124998 (+ 469us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:44:52.125484 (+ 486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:44:52.125487 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:44:52.125489 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:44:52.125678 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:44:52.126312 (+ 634us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:44:52.126318 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:44:52.126666 (+ 348us) client_negotiation.cc:769] Sending connection context
0926 03:44:52.126977 (+ 311us) client_negotiation.cc:240] Negotiation successful
0926 03:44:52.127412 (+ 435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":327,"threads_started":1}
I20260926 03:44:52.130280 9388 heartbeater.cc:347] Connected to a master server at 127.0.194.254:36233
I20260926 03:44:52.130616 9388 heartbeater.cc:464] Registering TS with master...
I20260926 03:44:52.131398 9388 heartbeater.cc:511] Master 127.0.194.254:36233 requested a full tablet report, sending...
I20260926 03:44:52.132966 9065 ts_manager.cc:194] Registered new tserver with Master: 32a1a2dbed28471d96e29ce6cb51b40f (127.0.194.194:42665)
WARNING: no policy specified for kudu/127.0.194.195@KRBTEST.COM; defaulting to no policy
I20260926 03:44:52.133777 9065 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.194.194@KRBTEST.COM'} at 127.0.194.194:43081
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.194.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.194.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790394221564982-779-0/krb5kdc/kudu_127.0.194.195.keytab.
Entry for principal kudu/127.0.194.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790394221564982-779-0/krb5kdc/kudu_127.0.194.195.keytab.
I20260926 03:44:52.161460 779 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
/tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790394221564982-779-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790394221564982-779-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790394221564982-779-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790394221564982-779-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.194.195:0
--local_ip_for_outbound_sockets=127.0.194.195
--webserver_interface=127.0.194.195
--webserver_port=0
--tserver_master_addrs=127.0.194.254:36233
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790394221564982-779-0/krb5kdc/kudu_127.0.194.195.keytab
--principal=kudu/127.0.194.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.194.212:46875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790394221564982-779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790394221564982-779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790394221564982-779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:44:52.485651 9393 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:44:52.485998 9393 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:44:52.486174 9393 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:44:52.493454 9393 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:44:52.493631 9393 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:44:52.493821 9393 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.194.195
I20260926 03:44:52.503759 9393 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.194.212:46875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790394221564982-779-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790394221564982-779-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790394221564982-779-0/krb5kdc/kudu_127.0.194.195.keytab
--principal=kudu/127.0.194.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.194.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790394221564982-779-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.194.195
--webserver_port=0
--tserver_master_addrs=127.0.194.254:36233
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.194.195
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790394221564982-779-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:44:52.505163 9393 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:44:52.506498 9393 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:44:52.519536 9398 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:44:52.519586 9401 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:44:52.519614 9399 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:44:52.520383 9393 server_base.cc:1034] running on GCE node
I20260926 03:44:52.521059 9393 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:44:52.522152 9393 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:44:52.523344 9393 hybrid_clock.cc:648] HybridClock initialized: now 1790394292523299 us; error 34 us; skew 500 ppm
I20260926 03:44:52.523692 9393 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:44:52 dist-test-slave-n2th krb5kdc[9020](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394292, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:44:52.526714 9393 init.cc:377] Logged in from keytab as kudu/127.0.194.195@KRBTEST.COM (short username kudu)
I20260926 03:44:52.528246 9393 webserver.cc:460] Webserver started at http://127.0.194.195:45631/ using document root <none> and password file <none>
I20260926 03:44:52.528937 9393 fs_manager.cc:359] Metadata directory not provided
I20260926 03:44:52.529076 9393 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:44:52.529340 9393 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:44:52.532395 9393 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790394221564982-779-0/minicluster-data/ts-2/data/instance:
uuid: "94d0ac51c7c140ac97c2859d4ea6435d"
format_stamp: "Formatted at 2026-09-26 03:44:52 on dist-test-slave-n2th"
server_key: "c2920b753deccc75e1176f445434e629"
server_key_iv: "0e402313fff9ab6a656054571519501a"
server_key_version: "encryptionkey@0"
I20260926 03:44:52.533109 9393 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790394221564982-779-0/minicluster-data/ts-2/wal/instance:
uuid: "94d0ac51c7c140ac97c2859d4ea6435d"
format_stamp: "Formatted at 2026-09-26 03:44:52 on dist-test-slave-n2th"
server_key: "c2920b753deccc75e1176f445434e629"
server_key_iv: "0e402313fff9ab6a656054571519501a"
server_key_version: "encryptionkey@0"
I20260926 03:44:52.537971 9393 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260926 03:44:52.541409 9408 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:44:52.542719 9393 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 03:44:52.542903 9393 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790394221564982-779-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790394221564982-779-0/minicluster-data/ts-2/wal
uuid: "94d0ac51c7c140ac97c2859d4ea6435d"
format_stamp: "Formatted at 2026-09-26 03:44:52 on dist-test-slave-n2th"
server_key: "c2920b753deccc75e1176f445434e629"
server_key_iv: "0e402313fff9ab6a656054571519501a"
server_key_version: "encryptionkey@0"
I20260926 03:44:52.543066 9393 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790394221564982-779-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790394221564982-779-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790394221564982-779-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:44:52.577621 9393 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:44:52.582026 9393 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:44:52.582309 9393 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:44:52.583313 9393 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:44:52.584667 9393 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:44:52.584743 9393 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:44:52.584849 9393 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:44:52.584894 9393 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:44:52.618889 9393 rpc_server.cc:307] RPC server started. Bound to: 127.0.194.195:45223
I20260926 03:44:52.618928 9520 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.194.195:45223 every 8 connection(s)
I20260926 03:44:52.620271 9393 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790394221564982-779-0/minicluster-data/ts-2/data/info.pb
I20260926 03:44:52.623486 779 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu as pid 9393
I20260926 03:44:52.623659 779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790394221564982-779-0/minicluster-data/ts-2/wal/instance
I20260926 03:44:52.624279 779 external_mini_cluster.cc:1428] Setting key e8b8215f17c6e65fcb3d456e7e1ecc03
I20260926 03:44:52.632448 9254 heartbeater.cc:503] Master 127.0.194.254:36233 was elected leader, sending a full tablet report...
Sep 26 03:44:52 dist-test-slave-n2th krb5kdc[9020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394292, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.195@KRBTEST.COM for kudu/127.0.194.254@KRBTEST.COM
I20260926 03:44:52.640760 9256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:52.622881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.195:48651 (local address 127.0.194.254:36233)
0926 03:44:52.623074 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0926 03:44:52.623084 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 03:44:52.624508 (+ 1424us) server_negotiation.cc:407] Connection header received
0926 03:44:52.626614 (+ 2106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:44:52.626620 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:44:52.626748 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:44:52.626837 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:44:52.628386 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:52.629241 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:52.631021 (+ 1780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:52.631431 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:52.631566 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:52.635267 (+ 3701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:44:52.635272 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:44:52.635275 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:44:52.635305 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:44:52.637621 (+ 2316us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:52.638441 (+ 820us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:52.638448 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:52.638457 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:52.638575 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:52.639104 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:52.639107 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:52.639109 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:52.639473 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:44:52.639600 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0926 03:44:52.640245 (+ 645us) server_negotiation.cc:299] Negotiation successful
0926 03:44:52.640456 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 03:44:52.641630 9523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:52.623546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:36233 (local address 127.0.194.195:48651)
0926 03:44:52.624308 (+ 762us) negotiation.cc:107] Waiting for socket to connect
0926 03:44:52.624392 (+ 84us) client_negotiation.cc:174] Beginning negotiation
0926 03:44:52.626361 (+ 1969us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:44:52.627133 (+ 772us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:44:52.627146 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:44:52.627675 (+ 529us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:44:52.628221 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:52.628236 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:52.629447 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:52.629451 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:52.630883 (+ 1432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:52.630894 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:52.631654 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:52.631664 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:52.631907 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:52.632617 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:44:52.632657 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:44:52.635058 (+ 2401us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:44:52.637769 (+ 2711us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:44:52.637777 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:44:52.637794 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:44:52.638315 (+ 521us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:44:52.638685 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:44:52.638691 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:44:52.638695 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:44:52.638988 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:44:52.639606 (+ 618us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:44:52.639612 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:44:52.639929 (+ 317us) client_negotiation.cc:769] Sending connection context
0926 03:44:52.640160 (+ 231us) client_negotiation.cc:240] Negotiation successful
0926 03:44:52.640539 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":331,"threads_started":1}
I20260926 03:44:52.643047 9521 heartbeater.cc:347] Connected to a master server at 127.0.194.254:36233
I20260926 03:44:52.643342 9521 heartbeater.cc:464] Registering TS with master...
I20260926 03:44:52.644076 9521 heartbeater.cc:511] Master 127.0.194.254:36233 requested a full tablet report, sending...
I20260926 03:44:52.645457 9065 ts_manager.cc:194] Registered new tserver with Master: 94d0ac51c7c140ac97c2859d4ea6435d (127.0.194.195:45223)
I20260926 03:44:52.646323 9065 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.194.195@KRBTEST.COM'} at 127.0.194.195:48651
I20260926 03:44:52.651598 779 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.194.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.194.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.194.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790394221564982-779-0/krb5kdc/oryx_127.0.194.193.keytab.
Entry for principal oryx/127.0.194.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790394221564982-779-0/krb5kdc/oryx_127.0.194.193.keytab.
WARNING: no policy specified for oryx/127.0.194.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.194.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.194.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790394221564982-779-0/krb5kdc/oryx_127.0.194.194.keytab.
Entry for principal oryx/127.0.194.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790394221564982-779-0/krb5kdc/oryx_127.0.194.194.keytab.
WARNING: no policy specified for oryx/127.0.194.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.194.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.194.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790394221564982-779-0/krb5kdc/oryx_127.0.194.195.keytab.
Entry for principal oryx/127.0.194.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790394221564982-779-0/krb5kdc/oryx_127.0.194.195.keytab.
I20260926 03:44:52.782013 779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu pid 9126
Sep 26 03:44:52 dist-test-slave-n2th krb5kdc[9020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394290, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.194.193@KRBTEST.COM
I20260926 03:44:52.797851 9531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:52.783172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45442 (local address 127.0.194.193:41821)
0926 03:44:52.783805 (+ 633us) server_negotiation.cc:206] Beginning negotiation
0926 03:44:52.783813 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:44:52.783839 (+ 26us) server_negotiation.cc:407] Connection header received
0926 03:44:52.783910 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:44:52.783917 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:44:52.784133 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:44:52.784274 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:44:52.785532 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:52.786562 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:52.787836 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:52.788384 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:52.788476 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:52.792419 (+ 3943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:44:52.792431 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:44:52.792445 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:44:52.792486 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:44:52.795587 (+ 3101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:52.796194 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:52.796201 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:52.796203 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:52.796325 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:52.796640 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:52.796643 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:52.796645 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:52.796970 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:44:52.797200 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0926 03:44:52.797383 (+ 183us) server_negotiation.cc:299] Negotiation successful
0926 03:44:52.797571 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":476,"thread_start_us":322,"threads_started":1}
I20260926 03:44:52.799162 9249 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:44:52.882656 779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu with pid 9126
I20260926 03:44:52.899158 779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu pid 9260
Sep 26 03:44:52 dist-test-slave-n2th krb5kdc[9020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394290, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.194.194@KRBTEST.COM
I20260926 03:44:52.914188 9533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:52.900027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44400 (local address 127.0.194.194:42665)
0926 03:44:52.900582 (+ 555us) server_negotiation.cc:206] Beginning negotiation
0926 03:44:52.900589 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:44:52.900614 (+ 25us) server_negotiation.cc:407] Connection header received
0926 03:44:52.900673 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:44:52.900679 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:44:52.900896 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:44:52.901029 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:44:52.902236 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:52.903321 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:52.904538 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:52.905077 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:52.905172 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:52.909249 (+ 4077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:44:52.909259 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:44:52.909274 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:44:52.909306 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:44:52.911900 (+ 2594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:52.912472 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:52.912479 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:52.912482 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:52.912598 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:52.912902 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:52.912905 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:52.912907 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:52.913205 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:44:52.913416 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0926 03:44:52.913683 (+ 267us) server_negotiation.cc:299] Negotiation successful
0926 03:44:52.913874 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":416,"thread_start_us":284,"threads_started":1}
I20260926 03:44:52.915640 9383 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:44:52.999264 779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu with pid 9260
I20260926 03:44:53.015363 779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu pid 9393
Sep 26 03:44:53 dist-test-slave-n2th krb5kdc[9020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394290, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.194.195@KRBTEST.COM
I20260926 03:44:53.032039 9535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:53.016521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59394 (local address 127.0.194.195:45223)
0926 03:44:53.017248 (+ 727us) server_negotiation.cc:206] Beginning negotiation
0926 03:44:53.017255 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:44:53.017292 (+ 37us) server_negotiation.cc:407] Connection header received
0926 03:44:53.017349 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:44:53.017355 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:44:53.017557 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:44:53.017733 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:44:53.019104 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:53.020128 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:53.021438 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:53.022007 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:53.022149 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:53.026465 (+ 4316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:44:53.026476 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:44:53.026492 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:44:53.026530 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:44:53.029235 (+ 2705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:53.029913 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:53.029921 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:53.029923 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:53.030051 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:53.030414 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:53.030419 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:53.030423 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:53.030829 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:44:53.031144 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0926 03:44:53.031430 (+ 286us) server_negotiation.cc:299] Negotiation successful
0926 03:44:53.031693 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":584,"thread_start_us":422,"threads_started":1}
I20260926 03:44:53.033315 9516 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:44:53.118710 779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu with pid 9393
I20260926 03:44:53.135152 779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu pid 9033
I20260926 03:44:53.136385 9095 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:44:53.237432 779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu with pid 9033
I20260926 03:44:53.258510 779 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
/tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790394221564982-779-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790394221564982-779-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790394221564982-779-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790394221564982-779-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.194.254:36233
--webserver_interface=127.0.194.254
--webserver_port=41557
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790394221564982-779-0/krb5kdc/kudu_127.0.194.254.keytab
--principal=kudu/127.0.194.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.194.212:46875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.194.254:36233
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790394221564982-779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790394221564982-779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790394221564982-779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:44:53.582929 9538 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:44:53.583240 9538 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:44:53.583418 9538 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:44:53.590924 9538 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:44:53.591042 9538 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:44:53.591135 9538 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:44:53.591229 9538 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:44:53.591296 9538 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:44:53.602234 9538 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.194.212:46875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790394221564982-779-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790394221564982-779-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.194.254:36233
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790394221564982-779-0/krb5kdc/kudu_127.0.194.254.keytab
--principal=kudu/127.0.194.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.194.254:36233
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790394221564982-779-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.194.254
--webserver_port=41557
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790394221564982-779-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:44:53.603668 9538 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:44:53.605083 9538 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:44:53.616684 9543 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:44:53.616706 9544 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:44:53.616685 9546 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:44:53.617468 9538 server_base.cc:1034] running on GCE node
I20260926 03:44:53.618269 9538 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:44:53.619896 9538 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:44:53.621134 9538 hybrid_clock.cc:648] HybridClock initialized: now 1790394293621083 us; error 74 us; skew 500 ppm
I20260926 03:44:53.621505 9538 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:44:53 dist-test-slave-n2th krb5kdc[9020](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394293, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:44:53.624723 9538 init.cc:377] Logged in from keytab as kudu/127.0.194.254@KRBTEST.COM (short username kudu)
I20260926 03:44:53.626151 9538 webserver.cc:460] Webserver started at http://127.0.194.254:41557/ using document root <none> and password file <none>
I20260926 03:44:53.626891 9538 fs_manager.cc:359] Metadata directory not provided
I20260926 03:44:53.627017 9538 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:44:53.632542 9538 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 03:44:53.635488 9553 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:44:53.636857 9538 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 03:44:53.637006 9538 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790394221564982-779-0/minicluster-data/master-0/data,/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790394221564982-779-0/minicluster-data/master-0/wal
uuid: "47aeac76c7fd4f84a64b43a172dfd695"
format_stamp: "Formatted at 2026-09-26 03:44:50 on dist-test-slave-n2th"
server_key: "dd019ebb6937c0c170a1140a9de40309"
server_key_iv: "1e4fc0e831863a411ac22fcf8b5c5d05"
server_key_version: "encryptionkey@0"
I20260926 03:44:53.637949 9538 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790394221564982-779-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790394221564982-779-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790394221564982-779-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:44:53.657748 9538 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:44:53.662405 9538 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:44:53.662632 9538 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:44:53.681485 9538 rpc_server.cc:307] RPC server started. Bound to: 127.0.194.254:36233
I20260926 03:44:53.681506 9604 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.194.254:36233 every 8 connection(s)
I20260926 03:44:53.683136 9538 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790394221564982-779-0/minicluster-data/master-0/data/info.pb
I20260926 03:44:53.690284 779 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu as pid 9538
I20260926 03:44:53.690392 9605 sys_catalog.cc:263] Verifying existing consensus state
I20260926 03:44:53.691455 779 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
/tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790394221564982-779-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790394221564982-779-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790394221564982-779-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790394221564982-779-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.194.193:41821
--local_ip_for_outbound_sockets=127.0.194.193
--tserver_master_addrs=127.0.194.254:36233
--webserver_port=43277
--webserver_interface=127.0.194.193
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790394221564982-779-0/krb5kdc/kudu_127.0.194.193.keytab
--principal=kudu/127.0.194.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.194.212:46875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.194.193
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790394221564982-779-0/krb5kdc/oryx_127.0.194.193.keytab with env {KRB5CCNAME=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790394221564982-779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790394221564982-779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790394221564982-779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 03:44:53.695638 9605 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47aeac76c7fd4f84a64b43a172dfd695: Bootstrap starting.
I20260926 03:44:53.717792 9605 log.cc:824] T 00000000000000000000000000000000 P 47aeac76c7fd4f84a64b43a172dfd695: Log is configured to *not* fsync() on all Append() calls
I20260926 03:44:53.726928 9605 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47aeac76c7fd4f84a64b43a172dfd695: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 03:44:53.727463 9605 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47aeac76c7fd4f84a64b43a172dfd695: Bootstrap complete.
I20260926 03:44:53.736562 9605 raft_consensus.cc:348] T 00000000000000000000000000000000 P 47aeac76c7fd4f84a64b43a172dfd695 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47aeac76c7fd4f84a64b43a172dfd695" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 36233 } }
I20260926 03:44:53.737422 9605 raft_consensus.cc:729] T 00000000000000000000000000000000 P 47aeac76c7fd4f84a64b43a172dfd695 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47aeac76c7fd4f84a64b43a172dfd695, State: Initialized, Role: FOLLOWER
I20260926 03:44:53.738088 9605 consensus_queue.cc:260] T 00000000000000000000000000000000 P 47aeac76c7fd4f84a64b43a172dfd695 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47aeac76c7fd4f84a64b43a172dfd695" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 36233 } }
I20260926 03:44:53.738361 9605 raft_consensus.cc:388] T 00000000000000000000000000000000 P 47aeac76c7fd4f84a64b43a172dfd695 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:44:53.738474 9605 raft_consensus.cc:482] T 00000000000000000000000000000000 P 47aeac76c7fd4f84a64b43a172dfd695 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:44:53.738667 9605 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 47aeac76c7fd4f84a64b43a172dfd695 [term 1 FOLLOWER]: Advancing to term 2
I20260926 03:44:53.742666 9605 raft_consensus.cc:504] T 00000000000000000000000000000000 P 47aeac76c7fd4f84a64b43a172dfd695 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47aeac76c7fd4f84a64b43a172dfd695" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 36233 } }
I20260926 03:44:53.743235 9605 leader_election.cc:302] T 00000000000000000000000000000000 P 47aeac76c7fd4f84a64b43a172dfd695 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 47aeac76c7fd4f84a64b43a172dfd695; no voters:
I20260926 03:44:53.743827 9605 leader_election.cc:288] T 00000000000000000000000000000000 P 47aeac76c7fd4f84a64b43a172dfd695 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 03:44:53.744050 9609 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 47aeac76c7fd4f84a64b43a172dfd695 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 03:44:53.745168 9609 raft_consensus.cc:686] T 00000000000000000000000000000000 P 47aeac76c7fd4f84a64b43a172dfd695 [term 2 LEADER]: Becoming Leader. State: Replica: 47aeac76c7fd4f84a64b43a172dfd695, State: Running, Role: LEADER
I20260926 03:44:53.745957 9605 sys_catalog.cc:564] T 00000000000000000000000000000000 P 47aeac76c7fd4f84a64b43a172dfd695 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:44:53.745934 9609 consensus_queue.cc:237] T 00000000000000000000000000000000 P 47aeac76c7fd4f84a64b43a172dfd695 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47aeac76c7fd4f84a64b43a172dfd695" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 36233 } }
I20260926 03:44:53.756938 9611 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47aeac76c7fd4f84a64b43a172dfd695 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 47aeac76c7fd4f84a64b43a172dfd695. Latest consensus state: current_term: 2 leader_uuid: "47aeac76c7fd4f84a64b43a172dfd695" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47aeac76c7fd4f84a64b43a172dfd695" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 36233 } } }
I20260926 03:44:53.757352 9610 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47aeac76c7fd4f84a64b43a172dfd695 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "47aeac76c7fd4f84a64b43a172dfd695" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47aeac76c7fd4f84a64b43a172dfd695" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 36233 } } }
I20260926 03:44:53.757568 9611 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47aeac76c7fd4f84a64b43a172dfd695 [sys.catalog]: This master's current role is: LEADER
I20260926 03:44:53.757634 9610 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47aeac76c7fd4f84a64b43a172dfd695 [sys.catalog]: This master's current role is: LEADER
I20260926 03:44:53.763252 9624 catalog_manager.cc:1259] Loaded cluster ID: fd2f1ea054f543fc90027c6c8e4a745d
I20260926 03:44:53.763372 9624 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 47aeac76c7fd4f84a64b43a172dfd695: loading cluster ID for follower catalog manager: success
I20260926 03:44:53.767318 9624 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 47aeac76c7fd4f84a64b43a172dfd695: acquiring CA information for follower catalog manager: success
I20260926 03:44:53.769721 9624 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 47aeac76c7fd4f84a64b43a172dfd695: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 03:44:53.769939 9626 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:44:53.772452 9626 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:44:53.773533 9626 catalog_manager.cc:1259] Loaded cluster ID: fd2f1ea054f543fc90027c6c8e4a745d
I20260926 03:44:53.773635 9626 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:44:53.775475 9626 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:44:53.776575 9626 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 47aeac76c7fd4f84a64b43a172dfd695: Loaded TSK: 0
I20260926 03:44:53.777535 9626 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 03:44:54.045596 9607 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:44:54.045943 9607 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:44:54.046123 9607 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:44:54.053920 9607 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:44:54.054100 9607 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:44:54.054333 9607 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.194.193
I20260926 03:44:54.064634 9607 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.194.212:46875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790394221564982-779-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790394221564982-779-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790394221564982-779-0/krb5kdc/oryx_127.0.194.193.keytab
--principal=oryx/127.0.194.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.194.193:41821
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790394221564982-779-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.194.193
--webserver_port=43277
--tserver_master_addrs=127.0.194.254:36233
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.194.193
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790394221564982-779-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:44:54.066061 9607 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:44:54.067474 9607 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:44:54.082194 9633 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:44:54.082408 9635 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:44:54.082238 9632 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:44:54.084108 9607 server_base.cc:1034] running on GCE node
I20260926 03:44:54.084892 9607 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:44:54.085918 9607 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:44:54.087129 9607 hybrid_clock.cc:648] HybridClock initialized: now 1790394294087076 us; error 47 us; skew 500 ppm
I20260926 03:44:54.087450 9607 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:44:54 dist-test-slave-n2th krb5kdc[9020](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394294, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.194.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:44:54.090570 9607 init.cc:377] Logged in from keytab as oryx/127.0.194.193@KRBTEST.COM (short username oryx)
I20260926 03:44:54.092211 9607 webserver.cc:460] Webserver started at http://127.0.194.193:43277/ using document root <none> and password file <none>
I20260926 03:44:54.092941 9607 fs_manager.cc:359] Metadata directory not provided
I20260926 03:44:54.093149 9607 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:44:54.098826 9607 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260926 03:44:54.101734 9642 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:44:54.103308 9607 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260926 03:44:54.103492 9607 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790394221564982-779-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790394221564982-779-0/minicluster-data/ts-0/wal
uuid: "4386950be0e24073a32223c354a87dd1"
format_stamp: "Formatted at 2026-09-26 03:44:51 on dist-test-slave-n2th"
server_key: "6e648b7fdce962d3d1b1c5f864ece6e9"
server_key_iv: "9d0b5c2c7e4ce14f8cb57486dea9c76d"
server_key_version: "encryptionkey@0"
I20260926 03:44:54.104519 9607 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790394221564982-779-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790394221564982-779-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790394221564982-779-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:44:54.128510 9607 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:44:54.132973 9607 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:44:54.133255 9607 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:44:54.134223 9607 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:44:54.135790 9607 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:44:54.135916 9607 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:44:54.136037 9607 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:44:54.136123 9607 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:44:54.168864 9607 rpc_server.cc:307] RPC server started. Bound to: 127.0.194.193:41821
I20260926 03:44:54.168951 9754 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.194.193:41821 every 8 connection(s)
I20260926 03:44:54.170296 9607 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790394221564982-779-0/minicluster-data/ts-0/data/info.pb
I20260926 03:44:54.176735 779 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu as pid 9607
I20260926 03:44:54.177435 779 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
/tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790394221564982-779-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790394221564982-779-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790394221564982-779-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790394221564982-779-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.194.194:42665
--local_ip_for_outbound_sockets=127.0.194.194
--tserver_master_addrs=127.0.194.254:36233
--webserver_port=46253
--webserver_interface=127.0.194.194
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790394221564982-779-0/krb5kdc/kudu_127.0.194.194.keytab
--principal=kudu/127.0.194.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.194.212:46875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.194.194
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790394221564982-779-0/krb5kdc/oryx_127.0.194.194.keytab with env {KRB5CCNAME=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790394221564982-779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790394221564982-779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790394221564982-779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 03:44:54 dist-test-slave-n2th krb5kdc[9020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.194.193@KRBTEST.COM for oryx/127.0.194.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:44:54 dist-test-slave-n2th krb5kdc[9020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.194.193@KRBTEST.COM for oryx/127.0.194.254@KRBTEST.COM, Server not found in Kerberos database
W20260926 03:44:54.545568 9759 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:44:54.545907 9759 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:44:54.546092 9759 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:44:54.553668 9759 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:44:54.553843 9759 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:44:54.554018 9759 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.194.194
I20260926 03:44:54.564069 9759 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.194.212:46875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790394221564982-779-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790394221564982-779-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790394221564982-779-0/krb5kdc/oryx_127.0.194.194.keytab
--principal=oryx/127.0.194.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.194.194:42665
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790394221564982-779-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.194.194
--webserver_port=46253
--tserver_master_addrs=127.0.194.254:36233
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.194.194
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790394221564982-779-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:44:54.565469 9759 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:44:54.566869 9759 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:44:54.585440 9766 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:44:54.585758 9765 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:44:54.586629 9768 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:44:54.588065 9759 server_base.cc:1034] running on GCE node
I20260926 03:44:54.588835 9759 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:44:54.589887 9759 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:44:54.593276 9759 hybrid_clock.cc:648] HybridClock initialized: now 1790394294593174 us; error 95 us; skew 500 ppm
I20260926 03:44:54.593732 9759 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:44:54 dist-test-slave-n2th krb5kdc[9020](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394294, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.194.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:44:54.597625 9759 init.cc:377] Logged in from keytab as oryx/127.0.194.194@KRBTEST.COM (short username oryx)
I20260926 03:44:54.599499 9759 webserver.cc:460] Webserver started at http://127.0.194.194:46253/ using document root <none> and password file <none>
I20260926 03:44:54.600243 9759 fs_manager.cc:359] Metadata directory not provided
I20260926 03:44:54.600427 9759 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:44:54.606283 9759 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.004s
I20260926 03:44:54.609607 9775 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:44:54.611480 9759 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.002s
I20260926 03:44:54.611672 9759 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790394221564982-779-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790394221564982-779-0/minicluster-data/ts-1/wal
uuid: "32a1a2dbed28471d96e29ce6cb51b40f"
format_stamp: "Formatted at 2026-09-26 03:44:52 on dist-test-slave-n2th"
server_key: "a11c4b3c3d25e319f706c2c2c920e56a"
server_key_iv: "a5151ec1e2965b5e39100e8c39632042"
server_key_version: "encryptionkey@0"
I20260926 03:44:54.612637 9759 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790394221564982-779-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790394221564982-779-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790394221564982-779-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:44:54.635720 9759 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:44:54.640882 9759 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:44:54.641187 9759 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:44:54.642306 9759 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:44:54.643839 9759 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:44:54.643970 9759 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:44:54.644102 9759 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:44:54.644187 9759 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:44:54.694515 9759 rpc_server.cc:307] RPC server started. Bound to: 127.0.194.194:42665
I20260926 03:44:54.694720 9887 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.194.194:42665 every 8 connection(s)
I20260926 03:44:54.696427 9759 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790394221564982-779-0/minicluster-data/ts-1/data/info.pb
I20260926 03:44:54.701030 779 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu as pid 9759
I20260926 03:44:54.701752 779 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
/tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790394221564982-779-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790394221564982-779-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790394221564982-779-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790394221564982-779-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.194.195:45223
--local_ip_for_outbound_sockets=127.0.194.195
--tserver_master_addrs=127.0.194.254:36233
--webserver_port=45631
--webserver_interface=127.0.194.195
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790394221564982-779-0/krb5kdc/kudu_127.0.194.195.keytab
--principal=kudu/127.0.194.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.194.212:46875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.194.195
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790394221564982-779-0/krb5kdc/oryx_127.0.194.195.keytab with env {KRB5CCNAME=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790394221564982-779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790394221564982-779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790394221564982-779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 03:44:54 dist-test-slave-n2th krb5kdc[9020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.194.194@KRBTEST.COM for oryx/127.0.194.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:44:54 dist-test-slave-n2th krb5kdc[9020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.194.194@KRBTEST.COM for oryx/127.0.194.254@KRBTEST.COM, Server not found in Kerberos database
W20260926 03:44:55.058822 9892 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:44:55.059154 9892 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:44:55.059326 9892 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:44:55.066645 9892 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:44:55.066836 9892 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:44:55.067018 9892 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.194.195
I20260926 03:44:55.077302 9892 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.194.212:46875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790394221564982-779-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790394221564982-779-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790394221564982-779-0/krb5kdc/oryx_127.0.194.195.keytab
--principal=oryx/127.0.194.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.194.195:45223
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790394221564982-779-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.194.195
--webserver_port=45631
--tserver_master_addrs=127.0.194.254:36233
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.194.195
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790394221564982-779-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:44:55.078900 9892 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:44:55.080209 9892 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:44:55.097267 9898 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:44:55.097579 9899 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:44:55.103675 9892 server_base.cc:1034] running on GCE node
W20260926 03:44:55.103891 9901 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:44:55.104606 9892 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:44:55.105813 9892 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:44:55.107030 9892 hybrid_clock.cc:648] HybridClock initialized: now 1790394295106956 us; error 65 us; skew 500 ppm
I20260926 03:44:55.107390 9892 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:44:55 dist-test-slave-n2th krb5kdc[9020](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394295, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.194.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:44:55.112078 9892 init.cc:377] Logged in from keytab as oryx/127.0.194.195@KRBTEST.COM (short username oryx)
I20260926 03:44:55.116362 9892 webserver.cc:460] Webserver started at http://127.0.194.195:45631/ using document root <none> and password file <none>
I20260926 03:44:55.117831 9892 fs_manager.cc:359] Metadata directory not provided
I20260926 03:44:55.118068 9892 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:44:55.129565 9892 fs_manager.cc:709] Time spent opening directory manager: real 0.007s user 0.005s sys 0.003s
I20260926 03:44:55.135344 9908 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:44:55.138178 9892 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.003s sys 0.001s
I20260926 03:44:55.138468 9892 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790394221564982-779-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790394221564982-779-0/minicluster-data/ts-2/wal
uuid: "94d0ac51c7c140ac97c2859d4ea6435d"
format_stamp: "Formatted at 2026-09-26 03:44:52 on dist-test-slave-n2th"
server_key: "c2920b753deccc75e1176f445434e629"
server_key_iv: "0e402313fff9ab6a656054571519501a"
server_key_version: "encryptionkey@0"
I20260926 03:44:55.140600 9892 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790394221564982-779-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790394221564982-779-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790394221564982-779-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:44:55.177002 9892 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:44:55.181545 9892 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:44:55.181823 9892 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:44:55.182930 9892 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:44:55.184322 9892 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:44:55.184433 9892 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:44:55.184558 9892 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:44:55.184666 9892 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:44:55.306128 9892 rpc_server.cc:307] RPC server started. Bound to: 127.0.194.195:45223
I20260926 03:44:55.306250 10020 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.194.195:45223 every 8 connection(s)
I20260926 03:44:55.308941 9892 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790394221564982-779-0/minicluster-data/ts-2/data/info.pb
I20260926 03:44:55.317431 779 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu as pid 9892
Sep 26 03:44:55 dist-test-slave-n2th krb5kdc[9020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.194.195@KRBTEST.COM for oryx/127.0.194.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:44:55 dist-test-slave-n2th krb5kdc[9020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.194.195@KRBTEST.COM for oryx/127.0.194.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:44:55.375485 10024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:55.318423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59358 (local address 127.0.194.254:36233)
0926 03:44:55.346128 (+ 27705us) server_negotiation.cc:206] Beginning negotiation
0926 03:44:55.346136 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:44:55.349277 (+ 3141us) server_negotiation.cc:407] Connection header received
0926 03:44:55.349351 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:44:55.349356 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:44:55.349431 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:44:55.349566 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:44:55.351829 (+ 2263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:55.352957 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:55.354858 (+ 1901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:55.355486 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:55.355656 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:55.360996 (+ 5340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:44:55.361014 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:44:55.361052 (+ 38us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:44:55.361122 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:44:55.367173 (+ 6051us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:55.368679 (+ 1506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:55.368695 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:55.368711 (+ 16us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:55.369062 (+ 351us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:44:55.369688 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:44:55.369697 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:44:55.369703 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:44:55.370704 (+ 1001us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:44:55.371191 (+ 487us) server_negotiation.cc:1035] Waiting for connection context
0926 03:44:55.373131 (+ 1940us) server_negotiation.cc:299] Negotiation successful
0926 03:44:55.373865 (+ 734us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27542,"thread_start_us":23119,"threads_started":1}
W20260926 03:44:55.675355 9758 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.492s user 0.001s sys 0.012s
W20260926 03:44:55.677641 9757 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:44:54.173381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.193:41303 (local address 127.0.194.254:36233)
0926 03:44:54.174039 (+ 658us) server_negotiation.cc:206] Beginning negotiation
0926 03:44:54.174050 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 03:44:54.174443 (+ 393us) server_negotiation.cc:407] Connection header received
0926 03:44:54.176604 (+ 2161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:44:54.176616 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:44:54.176996 (+ 380us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:44:54.177361 (+ 365us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:44:54.178843 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:54.180046 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:54.181569 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:54.182233 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:54.182425 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:55.677117 (+1494692us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.194.193:41303: BlockingRecv error: recv got EOF from 127.0.194.193:41303 (error 108)
Metrics: {"server-negotiator.queue_time_us":456,"thread_start_us":288,"threads_started":1}
I20260926 03:44:55.680480 9758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:54.173487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:36233 (local address 127.0.194.193:41303)
0926 03:44:54.174269 (+ 782us) negotiation.cc:107] Waiting for socket to connect
0926 03:44:54.174319 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 03:44:54.176053 (+ 1734us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:44:54.177540 (+ 1487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:44:54.177553 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:44:54.178122 (+ 569us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:44:54.178627 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:54.178643 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:54.180205 (+ 1562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:54.180215 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:54.181409 (+ 1194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:54.181420 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:54.182459 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:54.182466 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:54.182767 (+ 301us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:54.183445 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:44:54.183466 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:44:55.678834 (+1495368us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.194.254:36233: Server oryx/127.0.194.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":344,"threads_started":1}
W20260926 03:44:55.683636 9755 heartbeater.cc:644] Failed to heartbeat to 127.0.194.254:36233 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.194.254:36233: Client connection negotiation failed: client connection to 127.0.194.254:36233: Server oryx/127.0.194.254@KRBTEST.COM not found in Kerberos database
Sep 26 03:44:55 dist-test-slave-n2th krb5kdc[9020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.194.193@KRBTEST.COM for oryx/127.0.194.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:44:55 dist-test-slave-n2th krb5kdc[9020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.194.193@KRBTEST.COM for oryx/127.0.194.254@KRBTEST.COM, Server not found in Kerberos database
W20260926 03:44:55.707540 9757 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:44:55.686269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.193:55355 (local address 127.0.194.254:36233)
0926 03:44:55.686531 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0926 03:44:55.686542 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 03:44:55.688418 (+ 1876us) server_negotiation.cc:407] Connection header received
0926 03:44:55.688773 (+ 355us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:44:55.688781 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:44:55.688891 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:44:55.688986 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:44:55.690605 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:55.692515 (+ 1910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:55.694762 (+ 2247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:55.695438 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:55.695552 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:55.707161 (+ 11609us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.194.193:55355: BlockingRecv error: recv got EOF from 127.0.194.193:55355 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
I20260926 03:44:55.708489 9758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:55.686172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:36233 (local address 127.0.194.193:55355)
0926 03:44:55.688286 (+ 2114us) negotiation.cc:107] Waiting for socket to connect
0926 03:44:55.688317 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0926 03:44:55.688566 (+ 249us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:44:55.689319 (+ 753us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:44:55.689327 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:44:55.690044 (+ 717us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:44:55.690350 (+ 306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:55.690367 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:55.692862 (+ 2495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:55.692871 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:55.694543 (+ 1672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:55.694561 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:55.695889 (+ 1328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:55.695896 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:55.696088 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:55.700384 (+ 4296us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:44:55.700422 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:44:55.707997 (+ 7575us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.194.254:36233: Server oryx/127.0.194.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1914}
Sep 26 03:44:55 dist-test-slave-n2th krb5kdc[9020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.194.193@KRBTEST.COM for oryx/127.0.194.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:44:55 dist-test-slave-n2th krb5kdc[9020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.194.193@KRBTEST.COM for oryx/127.0.194.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:44:55.723604 10028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:55.709459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:36233 (local address 127.0.194.193:34747)
0926 03:44:55.710654 (+ 1195us) negotiation.cc:107] Waiting for socket to connect
0926 03:44:55.710682 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0926 03:44:55.710989 (+ 307us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:44:55.711713 (+ 724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:44:55.711722 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:44:55.712421 (+ 699us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:44:55.712758 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:55.712774 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:55.714459 (+ 1685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:55.714466 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:55.716294 (+ 1828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:55.716312 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:55.717513 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:55.717522 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:55.717735 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:55.718460 (+ 725us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:44:55.718483 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:44:55.723243 (+ 4760us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.194.254:36233: Server oryx/127.0.194.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":995,"thread_start_us":696,"threads_started":1}
W20260926 03:44:55.723713 9757 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:44:55.709710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.193:34747 (local address 127.0.194.254:36233)
0926 03:44:55.710011 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0926 03:44:55.710022 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 03:44:55.710833 (+ 811us) server_negotiation.cc:407] Connection header received
0926 03:44:55.711195 (+ 362us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:44:55.711204 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:44:55.711316 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:44:55.711418 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:44:55.712960 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:55.714246 (+ 1286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:55.716474 (+ 2228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:55.717110 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:55.717213 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:55.723464 (+ 6251us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.194.193:34747: BlockingRecv error: recv got EOF from 127.0.194.193:34747 (error 108)
Metrics: {"server-negotiator.queue_time_us":129}
W20260926 03:44:55.724148 9755 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 03:44:56.085879 9891 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.356s user 0.007s sys 0.012s
W20260926 03:44:56.087162 9890 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:44:54.700770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.194:54159 (local address 127.0.194.254:36233)
0926 03:44:54.716258 (+ 15488us) server_negotiation.cc:206] Beginning negotiation
0926 03:44:54.716263 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:44:54.716287 (+ 24us) server_negotiation.cc:407] Connection header received
0926 03:44:54.718423 (+ 2136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:44:54.718431 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:44:54.718530 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:44:54.718645 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:44:54.721786 (+ 3141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:54.723266 (+ 1480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:54.726906 (+ 3640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:54.727722 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:54.727896 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:56.086860 (+1358964us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.194.194:54159: BlockingRecv error: recv got EOF from 127.0.194.194:54159 (error 108)
Metrics: {"server-negotiator.queue_time_us":15340,"thread_start_us":15192,"threads_started":1}
I20260926 03:44:56.087706 9891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:54.701361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:36233 (local address 127.0.194.194:54159)
0926 03:44:54.713765 (+ 12404us) negotiation.cc:107] Waiting for socket to connect
0926 03:44:54.713854 (+ 89us) client_negotiation.cc:174] Beginning negotiation
0926 03:44:54.717879 (+ 4025us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:44:54.719137 (+ 1258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:44:54.719167 (+ 30us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:44:54.720353 (+ 1186us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:44:54.721553 (+ 1200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:54.721589 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:54.723480 (+ 1891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:54.723489 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:54.726625 (+ 3136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:54.726650 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:54.728051 (+ 1401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:54.728061 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:54.728529 (+ 468us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:54.729622 (+ 1093us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:44:54.729666 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:44:56.086638 (+1356972us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.194.254:36233: Server oryx/127.0.194.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":11876,"thread_start_us":11554,"threads_started":1}
W20260926 03:44:56.088577 9888 heartbeater.cc:644] Failed to heartbeat to 127.0.194.254:36233 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.194.254:36233: Client connection negotiation failed: client connection to 127.0.194.254:36233: Server oryx/127.0.194.254@KRBTEST.COM not found in Kerberos database
Sep 26 03:44:56 dist-test-slave-n2th krb5kdc[9020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.194.194@KRBTEST.COM for oryx/127.0.194.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:44:56 dist-test-slave-n2th krb5kdc[9020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.194.194@KRBTEST.COM for oryx/127.0.194.254@KRBTEST.COM, Server not found in Kerberos database
W20260926 03:44:56.105134 9890 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:44:56.089413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.194:44137 (local address 127.0.194.254:36233)
0926 03:44:56.089568 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0926 03:44:56.089572 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:44:56.089804 (+ 232us) server_negotiation.cc:407] Connection header received
0926 03:44:56.090002 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:44:56.090007 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:44:56.090078 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:44:56.090178 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:44:56.090947 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:56.091840 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:56.093218 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:56.093604 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:56.093677 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:56.104953 (+ 11276us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.194.194:44137: BlockingRecv error: recv got EOF from 127.0.194.194:44137 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 03:44:56.105269 9891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:56.089416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:36233 (local address 127.0.194.194:44137)
0926 03:44:56.089689 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0926 03:44:56.089704 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 03:44:56.089877 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:44:56.090203 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:44:56.090208 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:44:56.090573 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:44:56.090799 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:56.090809 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:56.091985 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:56.091991 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:56.093053 (+ 1062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:56.093068 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:56.093907 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:56.093911 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:56.094027 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:56.100460 (+ 6433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:44:56.100481 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:44:56.104749 (+ 4268us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.194.254:36233: Server oryx/127.0.194.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":171}
Sep 26 03:44:56 dist-test-slave-n2th krb5kdc[9020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.194.194@KRBTEST.COM for oryx/127.0.194.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:44:56 dist-test-slave-n2th krb5kdc[9020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.194.194@KRBTEST.COM for oryx/127.0.194.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:44:56.117132 9891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:56.106175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:36233 (local address 127.0.194.194:43545)
0926 03:44:56.106423 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0926 03:44:56.106453 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0926 03:44:56.106616 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:44:56.107074 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:44:56.107081 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:44:56.107762 (+ 681us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:44:56.108119 (+ 357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:56.108137 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:56.109440 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:56.109446 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:56.110714 (+ 1268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:56.110772 (+ 58us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:56.111484 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:56.111490 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:56.111680 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:56.112472 (+ 792us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:44:56.112492 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:44:56.116801 (+ 4309us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.194.254:36233: Server oryx/127.0.194.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":63}
W20260926 03:44:56.117477 9888 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 03:44:56.117702 9890 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:44:56.106372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.194:43545 (local address 127.0.194.254:36233)
0926 03:44:56.106527 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0926 03:44:56.106531 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:44:56.106547 (+ 16us) server_negotiation.cc:407] Connection header received
0926 03:44:56.106818 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:44:56.106821 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:44:56.106896 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:44:56.106979 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:44:56.108293 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:56.109257 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:56.110903 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:56.111323 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:56.111459 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:56.117103 (+ 5644us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.194.194:43545: BlockingRecv error: recv got EOF from 127.0.194.194:43545 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260926 03:44:56.550426 10023 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.197s user 0.001s sys 0.023s
W20260926 03:44:56.551687 10025 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:44:55.315102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.195:59135 (local address 127.0.194.254:36233)
0926 03:44:55.341035 (+ 25933us) server_negotiation.cc:206] Beginning negotiation
0926 03:44:55.341046 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 03:44:55.341080 (+ 34us) server_negotiation.cc:407] Connection header received
0926 03:44:55.341173 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:44:55.341181 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:44:55.341303 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:44:55.344897 (+ 3594us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:44:55.345212 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:55.347445 (+ 2233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:55.350626 (+ 3181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:55.351371 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:55.352431 (+ 1060us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:56.551482 (+1199051us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.194.195:59135: BlockingRecv error: recv got EOF from 127.0.194.195:59135 (error 108)
Metrics: {"server-negotiator.queue_time_us":25731,"thread_start_us":32751,"threads_started":1}
I20260926 03:44:56.552198 10023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:55.314220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:36233 (local address 127.0.194.195:59135)
0926 03:44:55.318259 (+ 4039us) negotiation.cc:107] Waiting for socket to connect
0926 03:44:55.318344 (+ 85us) client_negotiation.cc:174] Beginning negotiation
0926 03:44:55.322135 (+ 3791us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:44:55.342145 (+ 20010us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:44:55.342166 (+ 21us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:44:55.343343 (+ 1177us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:44:55.344492 (+ 1149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:55.344530 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:55.348076 (+ 3546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:55.348085 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:55.350422 (+ 2337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:55.350446 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:55.351640 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:55.351648 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:55.352004 (+ 356us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:55.352971 (+ 967us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:44:55.353008 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:44:56.551233 (+1198225us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.194.254:36233: Server oryx/127.0.194.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":3565,"thread_start_us":3216,"threads_started":1}
W20260926 03:44:56.553140 10021 heartbeater.cc:644] Failed to heartbeat to 127.0.194.254:36233 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.194.254:36233: Client connection negotiation failed: client connection to 127.0.194.254:36233: Server oryx/127.0.194.254@KRBTEST.COM not found in Kerberos database
Sep 26 03:44:56 dist-test-slave-n2th krb5kdc[9020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.194.195@KRBTEST.COM for oryx/127.0.194.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:44:56 dist-test-slave-n2th krb5kdc[9020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.194.195@KRBTEST.COM for oryx/127.0.194.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:44:56.562785 10023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:56.553873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:36233 (local address 127.0.194.195:57939)
0926 03:44:56.554081 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0926 03:44:56.554097 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 03:44:56.554199 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:44:56.554549 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:44:56.554552 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:44:56.554947 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:44:56.555112 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:56.555144 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:56.556453 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:56.556477 (+ 24us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:56.557570 (+ 1093us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:56.557584 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:56.558298 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:56.558302 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:56.558429 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:56.559161 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:44:56.559179 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:44:56.562515 (+ 3336us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.194.254:36233: Server oryx/127.0.194.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":68}
W20260926 03:44:56.563026 10025 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:44:56.553979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.195:57939 (local address 127.0.194.254:36233)
0926 03:44:56.554130 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0926 03:44:56.554134 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:44:56.554153 (+ 19us) server_negotiation.cc:407] Connection header received
0926 03:44:56.554349 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:44:56.554353 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:44:56.554417 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:44:56.554509 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:44:56.555292 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:56.556301 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:56.557696 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:56.558146 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:56.558245 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:56.562777 (+ 4532us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.194.195:57939: BlockingRecv error: recv got EOF from 127.0.194.195:57939 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 26 03:44:56 dist-test-slave-n2th krb5kdc[9020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.194.195@KRBTEST.COM for oryx/127.0.194.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:44:56 dist-test-slave-n2th krb5kdc[9020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.194.195@KRBTEST.COM for oryx/127.0.194.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:44:56.572355 10023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:56.563590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:36233 (local address 127.0.194.195:51533)
0926 03:44:56.563748 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0926 03:44:56.563766 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 03:44:56.563857 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:44:56.564217 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:44:56.564220 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:44:56.564518 (+ 298us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:44:56.564684 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:56.564692 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:56.565637 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:56.565640 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:56.566570 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:56.566581 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:56.567555 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:56.567562 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:56.567853 (+ 291us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:56.568524 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:44:56.568541 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:44:56.572145 (+ 3604us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.194.254:36233: Server oryx/127.0.194.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":52}
W20260926 03:44:56.572631 10025 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:44:56.563701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.195:51533 (local address 127.0.194.254:36233)
0926 03:44:56.563958 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0926 03:44:56.563962 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:44:56.563978 (+ 16us) server_negotiation.cc:407] Connection header received
0926 03:44:56.564024 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:44:56.564027 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:44:56.564093 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:44:56.564173 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:44:56.564805 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:56.565519 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:56.566733 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:56.567365 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:56.567494 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:56.572374 (+ 4880us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.194.195:51533: BlockingRecv error: recv got EOF from 127.0.194.195:51533 (error 108)
Metrics: {"server-negotiator.queue_time_us":163}
W20260926 03:44:56.572715 10021 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 26 03:44:56 dist-test-slave-n2th krb5kdc[9020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.194.193@KRBTEST.COM for oryx/127.0.194.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:44:56 dist-test-slave-n2th krb5kdc[9020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.194.193@KRBTEST.COM for oryx/127.0.194.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:44:56.733873 10029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:56.724771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:36233 (local address 127.0.194.193:45855)
0926 03:44:56.725382 (+ 611us) negotiation.cc:107] Waiting for socket to connect
0926 03:44:56.725405 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 03:44:56.725539 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:44:56.725990 (+ 451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:44:56.725995 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:44:56.726519 (+ 524us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:44:56.726761 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:56.726773 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:56.727860 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:56.727865 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:56.728788 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:56.728800 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:56.729430 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:56.729434 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:56.729551 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:56.730259 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:44:56.730277 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:44:56.733647 (+ 3370us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.194.254:36233: Server oryx/127.0.194.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":307,"threads_started":1}
W20260926 03:44:56.734225 10025 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:44:56.724971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.193:45855 (local address 127.0.194.254:36233)
0926 03:44:56.725193 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0926 03:44:56.725198 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:44:56.725512 (+ 314us) server_negotiation.cc:407] Connection header received
0926 03:44:56.725718 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:44:56.725721 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:44:56.725785 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:44:56.725907 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:44:56.726907 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:56.727702 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:56.728912 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:56.729285 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:56.729378 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:56.733970 (+ 4592us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.194.193:45855: BlockingRecv error: recv got EOF from 127.0.194.193:45855 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Sep 26 03:44:57 dist-test-slave-n2th krb5kdc[9020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.194.194@KRBTEST.COM for oryx/127.0.194.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:44:57 dist-test-slave-n2th krb5kdc[9020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.194.194@KRBTEST.COM for oryx/127.0.194.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:44:57.127079 10030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:57.118165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:36233 (local address 127.0.194.194:58177)
0926 03:44:57.118809 (+ 644us) negotiation.cc:107] Waiting for socket to connect
0926 03:44:57.118825 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 03:44:57.118965 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:44:57.119328 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:44:57.119332 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:44:57.119751 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:44:57.119897 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:57.119905 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:57.120924 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:57.120928 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:57.121890 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:57.121902 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:57.122646 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:57.122650 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:57.122808 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:57.123387 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:44:57.123404 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:44:57.126842 (+ 3438us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.194.254:36233: Server oryx/127.0.194.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":375,"threads_started":1}
W20260926 03:44:57.127388 10025 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:44:57.118247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.194:58177 (local address 127.0.194.254:36233)
0926 03:44:57.118427 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0926 03:44:57.118433 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:44:57.118904 (+ 471us) server_negotiation.cc:407] Connection header received
0926 03:44:57.119095 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:44:57.119100 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:44:57.119180 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:44:57.119288 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:44:57.120021 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:57.120782 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:57.122024 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:57.122476 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:57.122570 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:57.127069 (+ 4499us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.194.194:58177: BlockingRecv error: recv got EOF from 127.0.194.194:58177 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 26 03:44:57 dist-test-slave-n2th krb5kdc[9020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.194.195@KRBTEST.COM for oryx/127.0.194.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:44:57 dist-test-slave-n2th krb5kdc[9020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.194.195@KRBTEST.COM for oryx/127.0.194.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:44:57.582491 10031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:57.573416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:36233 (local address 127.0.194.195:51839)
0926 03:44:57.573921 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0926 03:44:57.573938 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 03:44:57.574067 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:44:57.574363 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:44:57.574366 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:44:57.574860 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:44:57.575004 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:57.575012 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:57.575974 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:57.575979 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:57.576970 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:57.576981 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:57.577703 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:57.577708 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:57.577843 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:57.578414 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:44:57.578431 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:44:57.582233 (+ 3802us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.194.254:36233: Server oryx/127.0.194.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":293,"threads_started":1}
W20260926 03:44:57.582631 10025 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:44:57.573488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.195:51839 (local address 127.0.194.254:36233)
0926 03:44:57.573718 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0926 03:44:57.573728 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 03:44:57.574031 (+ 303us) server_negotiation.cc:407] Connection header received
0926 03:44:57.574155 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:44:57.574158 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:44:57.574232 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:44:57.574317 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:44:57.575128 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:57.575832 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:57.577099 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:57.577510 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:57.577612 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:57.582449 (+ 4837us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.194.195:51839: BlockingRecv error: recv got EOF from 127.0.194.195:51839 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 26 03:44:57 dist-test-slave-n2th krb5kdc[9020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.194.193@KRBTEST.COM for oryx/127.0.194.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:44:57 dist-test-slave-n2th krb5kdc[9020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.194.193@KRBTEST.COM for oryx/127.0.194.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:44:57.743639 10032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:57.734953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:36233 (local address 127.0.194.193:52861)
0926 03:44:57.735480 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0926 03:44:57.735496 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 03:44:57.735609 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:44:57.735893 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:44:57.735897 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:44:57.736277 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:44:57.736426 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:57.736433 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:57.737398 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:57.737402 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:57.738368 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:57.738379 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:57.739209 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:57.739215 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:57.739348 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:57.740042 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:44:57.740059 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:44:57.743409 (+ 3350us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.194.254:36233: Server oryx/127.0.194.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":307,"threads_started":1}
W20260926 03:44:57.743886 10025 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:44:57.734981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.193:52861 (local address 127.0.194.254:36233)
0926 03:44:57.735162 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0926 03:44:57.735166 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:44:57.735569 (+ 403us) server_negotiation.cc:407] Connection header received
0926 03:44:57.735704 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:44:57.735707 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:44:57.735765 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:44:57.735849 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:44:57.736549 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:57.737239 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:57.738527 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:57.739047 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:57.739189 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:57.743636 (+ 4447us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.194.193:52861: BlockingRecv error: recv got EOF from 127.0.194.193:52861 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 26 03:44:58 dist-test-slave-n2th krb5kdc[9020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.194.194@KRBTEST.COM for oryx/127.0.194.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:44:58 dist-test-slave-n2th krb5kdc[9020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.194.194@KRBTEST.COM for oryx/127.0.194.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:44:58.137384 10033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:58.128364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:36233 (local address 127.0.194.194:56217)
0926 03:44:58.129040 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0926 03:44:58.129056 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 03:44:58.129205 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:44:58.129553 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:44:58.129557 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:44:58.129985 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:44:58.130188 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:58.130202 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:58.131211 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:58.131215 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:58.132176 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:58.132191 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:58.132929 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:58.132934 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:58.133059 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:58.133630 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:44:58.133648 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:44:58.137137 (+ 3489us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.194.254:36233: Server oryx/127.0.194.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":418,"threads_started":1}
W20260926 03:44:58.137650 10025 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:44:58.128561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.194:56217 (local address 127.0.194.254:36233)
0926 03:44:58.128818 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0926 03:44:58.128824 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:44:58.129119 (+ 295us) server_negotiation.cc:407] Connection header received
0926 03:44:58.129326 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:44:58.129329 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:44:58.129392 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:44:58.129471 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:44:58.130319 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:58.131043 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:58.132328 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:58.132749 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:58.132850 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:58.137410 (+ 4560us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.194.194:56217: BlockingRecv error: recv got EOF from 127.0.194.194:56217 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
Sep 26 03:44:58 dist-test-slave-n2th krb5kdc[9020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.194.195@KRBTEST.COM for oryx/127.0.194.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:44:58 dist-test-slave-n2th krb5kdc[9020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.194.195@KRBTEST.COM for oryx/127.0.194.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:44:58.593029 10034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:58.583601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:36233 (local address 127.0.194.195:39457)
0926 03:44:58.584212 (+ 611us) negotiation.cc:107] Waiting for socket to connect
0926 03:44:58.584235 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 03:44:58.584357 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:44:58.584674 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:44:58.584678 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:44:58.585064 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:44:58.585253 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:58.585262 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:58.586458 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:58.586463 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:58.587621 (+ 1158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:58.587633 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:58.588345 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:58.588351 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:58.588518 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:58.589145 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:44:58.589164 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:44:58.592665 (+ 3501us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.194.254:36233: Server oryx/127.0.194.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":368,"threads_started":1}
W20260926 03:44:58.593183 10025 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:44:58.583858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.195:39457 (local address 127.0.194.254:36233)
0926 03:44:58.584113 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0926 03:44:58.584118 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:44:58.584321 (+ 203us) server_negotiation.cc:407] Connection header received
0926 03:44:58.584445 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:44:58.584448 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:44:58.584509 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:44:58.584594 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:44:58.585423 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:58.586322 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:58.587758 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:58.588186 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:58.588316 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:58.592930 (+ 4614us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.194.195:39457: BlockingRecv error: recv got EOF from 127.0.194.195:39457 (error 108)
Metrics: {"server-negotiator.queue_time_us":127}
Sep 26 03:44:58 dist-test-slave-n2th krb5kdc[9020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.194.193@KRBTEST.COM for oryx/127.0.194.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:44:58 dist-test-slave-n2th krb5kdc[9020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.194.193@KRBTEST.COM for oryx/127.0.194.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:44:58.753882 10035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:58.744899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:36233 (local address 127.0.194.193:36481)
0926 03:44:58.745381 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0926 03:44:58.745398 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 03:44:58.745559 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:44:58.745887 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:44:58.745891 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:44:58.746282 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:44:58.746487 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:58.746502 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:58.747612 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:58.747620 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:58.748597 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:58.748608 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:58.749469 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:58.749475 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:58.749661 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:58.750270 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:44:58.750287 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:44:58.753650 (+ 3363us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.194.254:36233: Server oryx/127.0.194.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":283,"threads_started":1}
W20260926 03:44:58.754106 10025 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:44:58.744974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.193:36481 (local address 127.0.194.254:36233)
0926 03:44:58.745165 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0926 03:44:58.745172 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:44:58.745485 (+ 313us) server_negotiation.cc:407] Connection header received
0926 03:44:58.745678 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:44:58.745683 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:44:58.745757 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:44:58.745857 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:44:58.746635 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:58.747467 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:58.748768 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:58.749310 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:58.749460 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:58.753869 (+ 4409us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.194.193:36481: BlockingRecv error: recv got EOF from 127.0.194.193:36481 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Sep 26 03:44:59 dist-test-slave-n2th krb5kdc[9020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.194.194@KRBTEST.COM for oryx/127.0.194.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:44:59 dist-test-slave-n2th krb5kdc[9020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.194.194@KRBTEST.COM for oryx/127.0.194.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:44:59.147261 10036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:59.138353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:36233 (local address 127.0.194.194:59683)
0926 03:44:59.138985 (+ 632us) negotiation.cc:107] Waiting for socket to connect
0926 03:44:59.139002 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 03:44:59.139160 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:44:59.139521 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:44:59.139525 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:44:59.139915 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:44:59.140082 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:59.140091 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:59.141135 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:59.141142 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:59.142045 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:59.142056 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:59.142821 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:59.142829 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:59.143002 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:59.143578 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:44:59.143595 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:44:59.147036 (+ 3441us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.194.254:36233: Server oryx/127.0.194.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":532,"thread_start_us":396,"threads_started":1}
W20260926 03:44:59.147621 10025 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:44:59.138426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.194:59683 (local address 127.0.194.254:36233)
0926 03:44:59.138821 (+ 395us) server_negotiation.cc:206] Beginning negotiation
0926 03:44:59.138827 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:44:59.139088 (+ 261us) server_negotiation.cc:407] Connection header received
0926 03:44:59.139309 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:44:59.139314 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:44:59.139382 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:44:59.139487 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:44:59.140235 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:59.140999 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:59.142189 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:59.142608 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:59.142796 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:59.147247 (+ 4451us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.194.194:59683: BlockingRecv error: recv got EOF from 127.0.194.194:59683 (error 108)
Metrics: {"server-negotiator.queue_time_us":195}
Sep 26 03:44:59 dist-test-slave-n2th krb5kdc[9020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.194.195@KRBTEST.COM for oryx/127.0.194.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:44:59 dist-test-slave-n2th krb5kdc[9020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.194.195@KRBTEST.COM for oryx/127.0.194.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:44:59.603312 10037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:59.594232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:36233 (local address 127.0.194.195:52291)
0926 03:44:59.594706 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0926 03:44:59.594996 (+ 290us) client_negotiation.cc:174] Beginning negotiation
0926 03:44:59.595124 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:44:59.595411 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:44:59.595414 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:44:59.595770 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:44:59.595933 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:59.595941 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:59.596986 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:59.596990 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:59.598257 (+ 1267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:59.598273 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:59.599055 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:59.599064 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:59.599237 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:59.599799 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:44:59.599815 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:44:59.603082 (+ 3267us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.194.254:36233: Server oryx/127.0.194.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":258,"threads_started":1}
W20260926 03:44:59.603461 10025 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:44:59.594285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.195:52291 (local address 127.0.194.254:36233)
0926 03:44:59.594522 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0926 03:44:59.594528 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:44:59.595085 (+ 557us) server_negotiation.cc:407] Connection header received
0926 03:44:59.595221 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:44:59.595224 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:44:59.595284 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:44:59.595368 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:44:59.596053 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:59.596850 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:59.598401 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:59.598891 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:59.599025 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:59.603267 (+ 4242us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.194.195:52291: BlockingRecv error: recv got EOF from 127.0.194.195:52291 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Sep 26 03:44:59 dist-test-slave-n2th krb5kdc[9020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.194.193@KRBTEST.COM for oryx/127.0.194.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:44:59 dist-test-slave-n2th krb5kdc[9020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.194.193@KRBTEST.COM for oryx/127.0.194.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:44:59.763864 10038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:44:59.754961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:36233 (local address 127.0.194.193:47283)
0926 03:44:59.755487 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0926 03:44:59.755503 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 03:44:59.755663 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:44:59.755994 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:44:59.755998 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:44:59.756419 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:44:59.756619 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:59.756634 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:59.757801 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:59.757807 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:59.758870 (+ 1063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:44:59.758880 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:44:59.759559 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:44:59.759565 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:44:59.759740 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:59.760387 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:44:59.760404 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:44:59.763612 (+ 3208us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.194.254:36233: Server oryx/127.0.194.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":307,"threads_started":1}
W20260926 03:44:59.764089 10025 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:44:59.755011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.193:47283 (local address 127.0.194.254:36233)
0926 03:44:59.755234 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0926 03:44:59.755238 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:44:59.755586 (+ 348us) server_negotiation.cc:407] Connection header received
0926 03:44:59.755788 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:44:59.755793 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:44:59.755861 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:44:59.755962 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:44:59.756780 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:59.757655 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:59.759006 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:44:59.759415 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:44:59.759550 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:44:59.763798 (+ 4248us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.194.193:47283: BlockingRecv error: recv got EOF from 127.0.194.193:47283 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Sep 26 03:45:00 dist-test-slave-n2th krb5kdc[9020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.194.194@KRBTEST.COM for oryx/127.0.194.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:45:00 dist-test-slave-n2th krb5kdc[9020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.194.194@KRBTEST.COM for oryx/127.0.194.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:45:00.157749 10039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:45:00.148351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:36233 (local address 127.0.194.194:48967)
0926 03:45:00.149061 (+ 710us) negotiation.cc:107] Waiting for socket to connect
0926 03:45:00.149083 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 03:45:00.149288 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:45:00.149671 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:45:00.149675 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:45:00.150074 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:45:00.150237 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:45:00.150245 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:45:00.151317 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:45:00.151324 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:45:00.152310 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:45:00.152320 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:45:00.153084 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:45:00.153089 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:45:00.153274 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:45:00.153867 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:45:00.153886 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:45:00.157389 (+ 3503us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.194.254:36233: Server oryx/127.0.194.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":570,"thread_start_us":412,"threads_started":1}
W20260926 03:45:00.157819 10025 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:45:00.148377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.194:48967 (local address 127.0.194.254:36233)
0926 03:45:00.148606 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0926 03:45:00.148615 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:45:00.149152 (+ 537us) server_negotiation.cc:407] Connection header received
0926 03:45:00.149465 (+ 313us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:45:00.149470 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:45:00.149554 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:45:00.149645 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:45:00.150383 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:45:00.151192 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:45:00.152453 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:45:00.152932 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:45:00.153060 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:45:00.157605 (+ 4545us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.194.194:48967: BlockingRecv error: recv got EOF from 127.0.194.194:48967 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 26 03:45:00 dist-test-slave-n2th krb5kdc[9020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.194.195@KRBTEST.COM for oryx/127.0.194.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:45:00 dist-test-slave-n2th krb5kdc[9020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.194.195@KRBTEST.COM for oryx/127.0.194.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:45:00.613288 10040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:45:00.604303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:36233 (local address 127.0.194.195:52339)
0926 03:45:00.604915 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0926 03:45:00.604933 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 03:45:00.605050 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:45:00.605438 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:45:00.605442 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:45:00.605800 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:45:00.605960 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:45:00.605968 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:45:00.607098 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:45:00.607102 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:45:00.608119 (+ 1017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:45:00.608132 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:45:00.608838 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:45:00.608842 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:45:00.608970 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:45:00.609592 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:45:00.609608 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:45:00.613006 (+ 3398us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.194.254:36233: Server oryx/127.0.194.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":331,"threads_started":1}
W20260926 03:45:00.613544 10025 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:45:00.604461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.195:52339 (local address 127.0.194.254:36233)
0926 03:45:00.604634 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0926 03:45:00.604638 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:45:00.605037 (+ 399us) server_negotiation.cc:407] Connection header received
0926 03:45:00.605172 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:45:00.605176 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:45:00.605244 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:45:00.605336 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:45:00.606132 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:45:00.606961 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:45:00.608281 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:45:00.608695 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:45:00.608809 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:45:00.613255 (+ 4446us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.194.195:52339: BlockingRecv error: recv got EOF from 127.0.194.195:52339 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 26 03:45:00 dist-test-slave-n2th krb5kdc[9020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.194.193@KRBTEST.COM for oryx/127.0.194.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:45:00 dist-test-slave-n2th krb5kdc[9020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.194.193@KRBTEST.COM for oryx/127.0.194.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:45:00.773950 10041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:45:00.764784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:36233 (local address 127.0.194.193:43037)
0926 03:45:00.765284 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0926 03:45:00.765308 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 03:45:00.765484 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:45:00.765821 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:45:00.765826 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:45:00.766248 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:45:00.766392 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:45:00.766400 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:45:00.767451 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:45:00.767458 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:45:00.768419 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:45:00.768429 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:45:00.769141 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:45:00.769148 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:45:00.769327 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:45:00.769982 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:45:00.769999 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:45:00.773584 (+ 3585us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.194.254:36233: Server oryx/127.0.194.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":260,"threads_started":1}
W20260926 03:45:00.774147 10025 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:45:00.764906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.193:43037 (local address 127.0.194.254:36233)
0926 03:45:00.765148 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0926 03:45:00.765154 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:45:00.765402 (+ 248us) server_negotiation.cc:407] Connection header received
0926 03:45:00.765616 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:45:00.765621 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:45:00.765686 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:45:00.765789 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:45:00.766545 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:45:00.767319 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:45:00.768571 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:45:00.768985 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:45:00.769122 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:45:00.773848 (+ 4726us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.194.193:43037: BlockingRecv error: recv got EOF from 127.0.194.193:43037 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
Sep 26 03:45:01 dist-test-slave-n2th krb5kdc[9020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.194.194@KRBTEST.COM for oryx/127.0.194.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:45:01 dist-test-slave-n2th krb5kdc[9020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.194.194@KRBTEST.COM for oryx/127.0.194.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:45:01.173300 10042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:45:01.158897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:36233 (local address 127.0.194.194:49207)
0926 03:45:01.159468 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0926 03:45:01.159484 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 03:45:01.159646 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:45:01.159992 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:45:01.159996 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:45:01.160431 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:45:01.160578 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:45:01.160587 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:45:01.161645 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:45:01.161653 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:45:01.162582 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:45:01.162592 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:45:01.163374 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:45:01.163380 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:45:01.163558 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:45:01.169112 (+ 5554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:45:01.169137 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:45:01.172971 (+ 3834us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.194.254:36233: Server oryx/127.0.194.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":309,"threads_started":1}
W20260926 03:45:01.173435 10025 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:45:01.159145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.194:49207 (local address 127.0.194.254:36233)
0926 03:45:01.159393 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0926 03:45:01.159399 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:45:01.159572 (+ 173us) server_negotiation.cc:407] Connection header received
0926 03:45:01.159775 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:45:01.159780 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:45:01.159850 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:45:01.159954 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:45:01.160727 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:45:01.161503 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:45:01.162727 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:45:01.163191 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:45:01.163350 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:45:01.173168 (+ 9818us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.194.194:49207: BlockingRecv error: recv got EOF from 127.0.194.194:49207 (error 108)
Metrics: {"server-negotiator.queue_time_us":96}
Sep 26 03:45:01 dist-test-slave-n2th krb5kdc[9020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.194.195@KRBTEST.COM for oryx/127.0.194.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:45:01 dist-test-slave-n2th krb5kdc[9020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.194.195@KRBTEST.COM for oryx/127.0.194.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:45:01.623031 10043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:45:01.614160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:36233 (local address 127.0.194.195:49829)
0926 03:45:01.614674 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0926 03:45:01.614690 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 03:45:01.614867 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:45:01.615163 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:45:01.615166 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:45:01.615590 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:45:01.615733 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:45:01.615741 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:45:01.616753 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:45:01.616756 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:45:01.617822 (+ 1066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:45:01.617834 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:45:01.618513 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:45:01.618517 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:45:01.618653 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:45:01.619353 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:45:01.619371 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:45:01.622796 (+ 3425us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.194.254:36233: Server oryx/127.0.194.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":284,"threads_started":1}
W20260926 03:45:01.623227 10025 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:45:01.614247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.195:49829 (local address 127.0.194.254:36233)
0926 03:45:01.614418 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0926 03:45:01.614422 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:45:01.614779 (+ 357us) server_negotiation.cc:407] Connection header received
0926 03:45:01.614968 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:45:01.614971 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:45:01.615032 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:45:01.615113 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:45:01.615860 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:45:01.616637 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:45:01.617948 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:45:01.618346 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:45:01.618451 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:45:01.622998 (+ 4547us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.194.195:49829: BlockingRecv error: recv got EOF from 127.0.194.195:49829 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 26 03:45:01 dist-test-slave-n2th krb5kdc[9020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.194.193@KRBTEST.COM for oryx/127.0.194.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:45:01 dist-test-slave-n2th krb5kdc[9020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.194.193@KRBTEST.COM for oryx/127.0.194.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:45:01.783546 10044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:45:01.775034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:36233 (local address 127.0.194.193:32813)
0926 03:45:01.775551 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0926 03:45:01.775566 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 03:45:01.775675 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:45:01.775958 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:45:01.775962 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:45:01.776333 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:45:01.776476 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:45:01.776483 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:45:01.777460 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:45:01.777464 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:45:01.778384 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:45:01.778395 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:45:01.779133 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:45:01.779137 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:45:01.779272 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:45:01.779818 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:45:01.779835 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:45:01.783196 (+ 3361us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.194.254:36233: Server oryx/127.0.194.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":287,"threads_started":1}
W20260926 03:45:01.783725 10025 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:45:01.775126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.193:32813 (local address 127.0.194.254:36233)
0926 03:45:01.775313 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0926 03:45:01.775317 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:45:01.775639 (+ 322us) server_negotiation.cc:407] Connection header received
0926 03:45:01.775765 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:45:01.775768 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:45:01.775826 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:45:01.775906 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:45:01.776599 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:45:01.777348 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:45:01.778510 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:45:01.778972 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:45:01.779074 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:45:01.783453 (+ 4379us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.194.193:32813: BlockingRecv error: recv got EOF from 127.0.194.193:32813 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 26 03:45:02 dist-test-slave-n2th krb5kdc[9020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.194.194@KRBTEST.COM for oryx/127.0.194.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:45:02 dist-test-slave-n2th krb5kdc[9020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.194.194@KRBTEST.COM for oryx/127.0.194.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:45:02.183740 10045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:45:02.174453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:36233 (local address 127.0.194.194:39275)
0926 03:45:02.175018 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0926 03:45:02.175034 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 03:45:02.175204 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:45:02.175563 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:45:02.175568 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:45:02.176015 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:45:02.176175 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:45:02.176184 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:45:02.177216 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:45:02.177221 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:45:02.178298 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:45:02.178311 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:45:02.179139 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:45:02.179144 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:45:02.179329 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:45:02.179927 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:45:02.179943 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:45:02.183393 (+ 3450us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.194.254:36233: Server oryx/127.0.194.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":341,"threads_started":1}
W20260926 03:45:02.183900 10025 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:45:02.174552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.194:39275 (local address 127.0.194.254:36233)
0926 03:45:02.174775 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0926 03:45:02.174821 (+ 46us) server_negotiation.cc:399] Waiting for connection header
0926 03:45:02.175125 (+ 304us) server_negotiation.cc:407] Connection header received
0926 03:45:02.175352 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:45:02.175361 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:45:02.175422 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:45:02.175531 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:45:02.176355 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:45:02.177069 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:45:02.178446 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:45:02.178976 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:45:02.179116 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:45:02.183633 (+ 4517us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.194.194:39275: BlockingRecv error: recv got EOF from 127.0.194.194:39275 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 26 03:45:02 dist-test-slave-n2th krb5kdc[9020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.194.195@KRBTEST.COM for oryx/127.0.194.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:45:02 dist-test-slave-n2th krb5kdc[9020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.194.195@KRBTEST.COM for oryx/127.0.194.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:45:02.632795 10046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:45:02.623997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:36233 (local address 127.0.194.195:41725)
0926 03:45:02.624501 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0926 03:45:02.624524 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 03:45:02.624638 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:45:02.624957 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:45:02.624960 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:45:02.625335 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:45:02.625493 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:45:02.625502 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:45:02.626440 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:45:02.626443 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:45:02.627422 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:45:02.627434 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:45:02.628149 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:45:02.628155 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:45:02.628312 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:45:02.628948 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:45:02.628964 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:45:02.632567 (+ 3603us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.194.254:36233: Server oryx/127.0.194.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":295,"threads_started":1}
W20260926 03:45:02.633023 10025 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:45:02.624156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.195:41725 (local address 127.0.194.254:36233)
0926 03:45:02.624324 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0926 03:45:02.624329 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:45:02.624600 (+ 271us) server_negotiation.cc:407] Connection header received
0926 03:45:02.624735 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:45:02.624739 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:45:02.624809 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:45:02.624898 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:45:02.625617 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:45:02.626328 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:45:02.627558 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:45:02.627959 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:45:02.628062 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:45:02.632771 (+ 4709us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.194.195:41725: BlockingRecv error: recv got EOF from 127.0.194.195:41725 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 26 03:45:02 dist-test-slave-n2th krb5kdc[9020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.194.193@KRBTEST.COM for oryx/127.0.194.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:45:02 dist-test-slave-n2th krb5kdc[9020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.194.193@KRBTEST.COM for oryx/127.0.194.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:45:02.793581 10047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:45:02.784722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:36233 (local address 127.0.194.193:58205)
0926 03:45:02.785288 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0926 03:45:02.785305 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 03:45:02.785430 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:45:02.785745 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:45:02.785749 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:45:02.786134 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:45:02.786313 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:45:02.786321 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:45:02.787392 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:45:02.787396 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:45:02.788336 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:45:02.788347 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:45:02.789301 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:45:02.789306 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:45:02.789431 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:45:02.789996 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:45:02.790012 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:45:02.793317 (+ 3305us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.194.254:36233: Server oryx/127.0.194.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":259,"threads_started":1}
W20260926 03:45:02.793716 10025 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:45:02.784860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.193:58205 (local address 127.0.194.254:36233)
0926 03:45:02.785148 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0926 03:45:02.785154 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:45:02.785395 (+ 241us) server_negotiation.cc:407] Connection header received
0926 03:45:02.785534 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:45:02.785537 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:45:02.785614 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:45:02.785701 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:45:02.786440 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:45:02.787244 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:45:02.788476 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:45:02.789100 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:45:02.789224 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:45:02.793516 (+ 4292us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.194.193:58205: BlockingRecv error: recv got EOF from 127.0.194.193:58205 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Sep 26 03:45:03 dist-test-slave-n2th krb5kdc[9020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.194.194@KRBTEST.COM for oryx/127.0.194.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:45:03 dist-test-slave-n2th krb5kdc[9020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.194.194@KRBTEST.COM for oryx/127.0.194.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:45:03.193950 10048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:45:03.184777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:36233 (local address 127.0.194.194:38515)
0926 03:45:03.185377 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0926 03:45:03.185393 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 03:45:03.185575 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:45:03.185953 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:45:03.185958 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:45:03.186367 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:45:03.186554 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:45:03.186563 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:45:03.187675 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:45:03.187680 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:45:03.188642 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:45:03.188654 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:45:03.189406 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:45:03.189413 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:45:03.189581 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:45:03.190231 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:45:03.190248 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:45:03.193685 (+ 3437us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.194.254:36233: Server oryx/127.0.194.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":306,"threads_started":1}
W20260926 03:45:03.194132 10025 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:45:03.184888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.194:38515 (local address 127.0.194.254:36233)
0926 03:45:03.185117 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0926 03:45:03.185126 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:45:03.185497 (+ 371us) server_negotiation.cc:407] Connection header received
0926 03:45:03.185712 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:45:03.185722 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:45:03.185782 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:45:03.185893 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:45:03.186710 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:45:03.187515 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:45:03.188799 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:45:03.189238 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:45:03.189372 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:45:03.193887 (+ 4515us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.194.194:38515: BlockingRecv error: recv got EOF from 127.0.194.194:38515 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 26 03:45:03 dist-test-slave-n2th krb5kdc[9020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.194.195@KRBTEST.COM for oryx/127.0.194.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:45:03 dist-test-slave-n2th krb5kdc[9020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.194.195@KRBTEST.COM for oryx/127.0.194.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:45:03.642861 10049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:45:03.633888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:36233 (local address 127.0.194.195:56547)
0926 03:45:03.634513 (+ 625us) negotiation.cc:107] Waiting for socket to connect
0926 03:45:03.634534 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 03:45:03.634690 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:45:03.635068 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:45:03.635073 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:45:03.635485 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:45:03.635657 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:45:03.635665 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:45:03.636650 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:45:03.636656 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:45:03.637618 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:45:03.637630 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:45:03.638352 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:45:03.638358 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:45:03.638529 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:45:03.639131 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:45:03.639147 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:45:03.642550 (+ 3403us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.194.254:36233: Server oryx/127.0.194.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":279,"threads_started":1}
W20260926 03:45:03.643151 10025 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:45:03.633931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.195:56547 (local address 127.0.194.254:36233)
0926 03:45:03.634129 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0926 03:45:03.634137 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:45:03.634605 (+ 468us) server_negotiation.cc:407] Connection header received
0926 03:45:03.634857 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:45:03.634862 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:45:03.634933 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:45:03.635037 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:45:03.635797 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:45:03.636515 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:45:03.637767 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:45:03.638195 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:45:03.638328 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:45:03.642880 (+ 4552us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.194.195:56547: BlockingRecv error: recv got EOF from 127.0.194.195:56547 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Sep 26 03:45:03 dist-test-slave-n2th krb5kdc[9020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.194.193@KRBTEST.COM for oryx/127.0.194.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:45:03 dist-test-slave-n2th krb5kdc[9020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.194.193@KRBTEST.COM for oryx/127.0.194.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:45:03.802959 10050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:45:03.794652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:36233 (local address 127.0.194.193:34073)
0926 03:45:03.795228 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0926 03:45:03.795245 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 03:45:03.795356 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:45:03.795631 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:45:03.795635 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:45:03.796024 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:45:03.796191 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:45:03.796199 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:45:03.797118 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:45:03.797122 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:45:03.798031 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:45:03.798042 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:45:03.798725 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:45:03.798780 (+ 55us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:45:03.798985 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:45:03.799580 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:45:03.799597 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:45:03.802656 (+ 3059us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.194.254:36233: Server oryx/127.0.194.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":325,"threads_started":1}
W20260926 03:45:03.803090 10025 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:45:03.794876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.193:34073 (local address 127.0.194.254:36233)
0926 03:45:03.795092 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0926 03:45:03.795098 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:45:03.795321 (+ 223us) server_negotiation.cc:407] Connection header received
0926 03:45:03.795437 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:45:03.795440 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:45:03.795513 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:45:03.795593 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:45:03.796306 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:45:03.797003 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:45:03.798158 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:45:03.798577 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:45:03.798670 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:45:03.802858 (+ 4188us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.194.193:34073: BlockingRecv error: recv got EOF from 127.0.194.193:34073 (error 108)
Metrics: {"server-negotiator.queue_time_us":103}
Sep 26 03:45:04 dist-test-slave-n2th krb5kdc[9020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.194.194@KRBTEST.COM for oryx/127.0.194.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:45:04 dist-test-slave-n2th krb5kdc[9020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.194.194@KRBTEST.COM for oryx/127.0.194.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:45:04.204432 10051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:45:04.195138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:36233 (local address 127.0.194.194:58467)
0926 03:45:04.195741 (+ 603us) negotiation.cc:107] Waiting for socket to connect
0926 03:45:04.195760 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 03:45:04.195947 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:45:04.196322 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:45:04.196327 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:45:04.196741 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:45:04.196898 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:45:04.196906 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:45:04.198071 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:45:04.198076 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:45:04.199089 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:45:04.199100 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:45:04.199984 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:45:04.199990 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:45:04.200152 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:45:04.200708 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:45:04.200725 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:45:04.204078 (+ 3353us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.194.254:36233: Server oryx/127.0.194.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":310,"threads_started":1}
W20260926 03:45:04.204547 10025 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:45:04.195204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.194:58467 (local address 127.0.194.254:36233)
0926 03:45:04.195429 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0926 03:45:04.195436 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:45:04.195843 (+ 407us) server_negotiation.cc:407] Connection header received
0926 03:45:04.196105 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:45:04.196109 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:45:04.196182 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:45:04.196290 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:45:04.197041 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:45:04.197915 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:45:04.199233 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:45:04.199780 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:45:04.199948 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:45:04.204295 (+ 4347us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.194.194:58467: BlockingRecv error: recv got EOF from 127.0.194.194:58467 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 26 03:45:04 dist-test-slave-n2th krb5kdc[9020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.194.195@KRBTEST.COM for oryx/127.0.194.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:45:04 dist-test-slave-n2th krb5kdc[9020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.194.195@KRBTEST.COM for oryx/127.0.194.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:45:04.653741 10052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:45:04.643749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:36233 (local address 127.0.194.195:54351)
0926 03:45:04.644315 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0926 03:45:04.644331 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 03:45:04.644443 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:45:04.644799 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:45:04.644803 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:45:04.645291 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:45:04.645480 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:45:04.645489 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:45:04.646612 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:45:04.646616 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:45:04.647742 (+ 1126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:45:04.647755 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:45:04.648615 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:45:04.648619 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:45:04.648730 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:45:04.649446 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:45:04.649469 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:45:04.653358 (+ 3889us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.194.254:36233: Server oryx/127.0.194.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":313,"threads_started":1}
W20260926 03:45:04.653810 10025 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:45:04.643778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.195:54351 (local address 127.0.194.254:36233)
0926 03:45:04.643938 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0926 03:45:04.643942 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:45:04.644402 (+ 460us) server_negotiation.cc:407] Connection header received
0926 03:45:04.644536 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:45:04.644539 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:45:04.644599 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:45:04.644682 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:45:04.645645 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:45:04.646479 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:45:04.647927 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:45:04.648416 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:45:04.648576 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:45:04.653613 (+ 5037us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.194.195:54351: BlockingRecv error: recv got EOF from 127.0.194.195:54351 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 26 03:45:04 dist-test-slave-n2th krb5kdc[9020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.194.193@KRBTEST.COM for oryx/127.0.194.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:45:04 dist-test-slave-n2th krb5kdc[9020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.194.193@KRBTEST.COM for oryx/127.0.194.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:45:04.813145 10053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:45:04.803930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:36233 (local address 127.0.194.193:56327)
0926 03:45:04.804462 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0926 03:45:04.804478 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 03:45:04.804591 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:45:04.804916 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:45:04.804919 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:45:04.805328 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:45:04.805470 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:45:04.805479 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:45:04.806424 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:45:04.806428 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:45:04.807443 (+ 1015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:45:04.807454 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:45:04.808184 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:45:04.808189 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:45:04.808324 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:45:04.809132 (+ 808us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:45:04.809149 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:45:04.812904 (+ 3755us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.194.254:36233: Server oryx/127.0.194.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":298,"threads_started":1}
W20260926 03:45:04.813400 10025 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:45:04.804028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.193:56327 (local address 127.0.194.254:36233)
0926 03:45:04.804198 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0926 03:45:04.804203 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:45:04.804554 (+ 351us) server_negotiation.cc:407] Connection header received
0926 03:45:04.804687 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:45:04.804691 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:45:04.804765 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:45:04.804861 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:45:04.805590 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:45:04.806301 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:45:04.807598 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:45:04.808055 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:45:04.808146 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:45:04.813143 (+ 4997us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.194.193:56327: BlockingRecv error: recv got EOF from 127.0.194.193:56327 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 26 03:45:05 dist-test-slave-n2th krb5kdc[9020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.194.194@KRBTEST.COM for oryx/127.0.194.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:45:05 dist-test-slave-n2th krb5kdc[9020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.194.194@KRBTEST.COM for oryx/127.0.194.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:45:05.220839 10054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:45:05.205530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:36233 (local address 127.0.194.194:37823)
0926 03:45:05.205990 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0926 03:45:05.206008 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 03:45:05.206124 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:45:05.206484 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:45:05.206487 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:45:05.206944 (+ 457us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:45:05.207088 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:45:05.207097 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:45:05.208415 (+ 1318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:45:05.208419 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:45:05.209457 (+ 1038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:45:05.209469 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:45:05.210295 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:45:05.210300 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:45:05.210425 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:45:05.217107 (+ 6682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:45:05.217127 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:45:05.220608 (+ 3481us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.194.254:36233: Server oryx/127.0.194.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":232,"threads_started":1}
W20260926 03:45:05.221060 10025 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:45:05.205614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.194:37823 (local address 127.0.194.254:36233)
0926 03:45:05.205838 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0926 03:45:05.205844 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:45:05.206096 (+ 252us) server_negotiation.cc:407] Connection header received
0926 03:45:05.206260 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:45:05.206268 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:45:05.206352 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:45:05.206432 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:45:05.207247 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:45:05.208283 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:45:05.209612 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:45:05.210097 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:45:05.210254 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:45:05.220807 (+ 10553us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.194.194:37823: BlockingRecv error: recv got EOF from 127.0.194.194:37823 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Sep 26 03:45:05 dist-test-slave-n2th krb5kdc[9020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.194.195@KRBTEST.COM for oryx/127.0.194.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:45:05 dist-test-slave-n2th krb5kdc[9020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.194.195@KRBTEST.COM for oryx/127.0.194.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:45:05.664184 10055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:45:05.654939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:36233 (local address 127.0.194.195:39775)
0926 03:45:05.655470 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0926 03:45:05.655486 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 03:45:05.655644 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:45:05.656004 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:45:05.656010 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:45:05.656440 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:45:05.656586 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:45:05.656594 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:45:05.657561 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:45:05.657566 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:45:05.658528 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:45:05.658539 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:45:05.659355 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:45:05.659361 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:45:05.659549 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:45:05.660171 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:45:05.660188 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:45:05.663844 (+ 3656us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.194.254:36233: Server oryx/127.0.194.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":272,"threads_started":1}
W20260926 03:45:05.664289 10025 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:45:05.654975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.195:39775 (local address 127.0.194.254:36233)
0926 03:45:05.655184 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0926 03:45:05.655190 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:45:05.655587 (+ 397us) server_negotiation.cc:407] Connection header received
0926 03:45:05.655775 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:45:05.655780 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:45:05.655851 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:45:05.655962 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:45:05.656730 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:45:05.657427 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:45:05.658673 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:45:05.659197 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:45:05.659339 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:45:05.664068 (+ 4729us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.194.195:39775: BlockingRecv error: recv got EOF from 127.0.194.195:39775 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 26 03:45:05 dist-test-slave-n2th krb5kdc[9020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.194.193@KRBTEST.COM for oryx/127.0.194.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:45:05 dist-test-slave-n2th krb5kdc[9020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.194.193@KRBTEST.COM for oryx/127.0.194.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:45:05.823308 10056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:45:05.814168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:36233 (local address 127.0.194.193:33075)
0926 03:45:05.814738 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0926 03:45:05.814785 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 03:45:05.814973 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:45:05.815328 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:45:05.815333 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:45:05.815829 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:45:05.815992 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:45:05.816001 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:45:05.817089 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:45:05.817095 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:45:05.818075 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:45:05.818085 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:45:05.818810 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:45:05.818816 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:45:05.818996 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:45:05.819647 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:45:05.819664 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:45:05.823054 (+ 3390us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.194.254:36233: Server oryx/127.0.194.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":327,"threads_started":1}
W20260926 03:45:05.823513 10025 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:45:05.814293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.193:33075 (local address 127.0.194.254:36233)
0926 03:45:05.814530 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0926 03:45:05.814538 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:45:05.814917 (+ 379us) server_negotiation.cc:407] Connection header received
0926 03:45:05.815109 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:45:05.815116 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:45:05.815195 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:45:05.815298 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:45:05.816157 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:45:05.816942 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:45:05.818214 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:45:05.818631 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:45:05.818788 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:45:05.823243 (+ 4455us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.194.193:33075: BlockingRecv error: recv got EOF from 127.0.194.193:33075 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 26 03:45:06 dist-test-slave-n2th krb5kdc[9020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.194.194@KRBTEST.COM for oryx/127.0.194.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:45:06 dist-test-slave-n2th krb5kdc[9020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.194.194@KRBTEST.COM for oryx/127.0.194.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:45:06.236869 10057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:45:06.222085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:36233 (local address 127.0.194.194:57561)
0926 03:45:06.222638 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0926 03:45:06.222658 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 03:45:06.222828 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:45:06.223264 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:45:06.223268 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:45:06.223797 (+ 529us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:45:06.223970 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:45:06.223980 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:45:06.224960 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:45:06.224964 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:45:06.225934 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:45:06.225945 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:45:06.226645 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:45:06.226649 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:45:06.226844 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:45:06.232522 (+ 5678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:45:06.232546 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:45:06.236555 (+ 4009us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.194.254:36233: Server oryx/127.0.194.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":280,"threads_started":1}
W20260926 03:45:06.237056 10025 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:45:06.222195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.194:57561 (local address 127.0.194.254:36233)
0926 03:45:06.222426 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0926 03:45:06.222434 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:45:06.222755 (+ 321us) server_negotiation.cc:407] Connection header received
0926 03:45:06.222990 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:45:06.222995 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:45:06.223097 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:45:06.223208 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:45:06.224114 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:45:06.224805 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:45:06.226095 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:45:06.226511 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:45:06.226615 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:45:06.236789 (+ 10174us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.194.194:57561: BlockingRecv error: recv got EOF from 127.0.194.194:57561 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 26 03:45:06 dist-test-slave-n2th krb5kdc[9020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.194.195@KRBTEST.COM for oryx/127.0.194.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:45:06 dist-test-slave-n2th krb5kdc[9020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.194.195@KRBTEST.COM for oryx/127.0.194.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:45:06.674613 10058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:45:06.665397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:36233 (local address 127.0.194.195:39519)
0926 03:45:06.666160 (+ 763us) negotiation.cc:107] Waiting for socket to connect
0926 03:45:06.666177 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 03:45:06.666326 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:45:06.666733 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:45:06.666759 (+ 26us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:45:06.667138 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:45:06.667359 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:45:06.667372 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:45:06.668518 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:45:06.668522 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:45:06.669531 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:45:06.669543 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:45:06.670214 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:45:06.670217 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:45:06.670329 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:45:06.671091 (+ 762us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:45:06.671109 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:45:06.674354 (+ 3245us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.194.254:36233: Server oryx/127.0.194.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":639,"thread_start_us":525,"threads_started":1}
W20260926 03:45:06.674916 10025 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:45:06.665604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.195:39519 (local address 127.0.194.254:36233)
0926 03:45:06.665830 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0926 03:45:06.665837 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:45:06.666245 (+ 408us) server_negotiation.cc:407] Connection header received
0926 03:45:06.666481 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:45:06.666486 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:45:06.666565 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:45:06.666674 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:45:06.667536 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:45:06.668374 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:45:06.669656 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:45:06.670084 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:45:06.670177 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:45:06.674594 (+ 4417us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.194.195:39519: BlockingRecv error: recv got EOF from 127.0.194.195:39519 (error 108)
Metrics: {"server-negotiator.queue_time_us":106}
Sep 26 03:45:06 dist-test-slave-n2th krb5kdc[9020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.194.193@KRBTEST.COM for oryx/127.0.194.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:45:06 dist-test-slave-n2th krb5kdc[9020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.194.193@KRBTEST.COM for oryx/127.0.194.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:45:06.835031 10059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:45:06.824460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:36233 (local address 127.0.194.193:54837)
0926 03:45:06.825197 (+ 737us) negotiation.cc:107] Waiting for socket to connect
0926 03:45:06.825219 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 03:45:06.825370 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:45:06.825821 (+ 451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:45:06.825830 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:45:06.826430 (+ 600us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:45:06.826880 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:45:06.826901 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:45:06.828051 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:45:06.828055 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:45:06.829623 (+ 1568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:45:06.829642 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:45:06.830466 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:45:06.830473 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:45:06.830643 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:45:06.831286 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:45:06.831302 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:45:06.834795 (+ 3493us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.194.254:36233: Server oryx/127.0.194.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":547,"thread_start_us":438,"threads_started":1}
W20260926 03:45:06.835403 10025 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:45:06.824681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.193:54837 (local address 127.0.194.254:36233)
0926 03:45:06.824943 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0926 03:45:06.824949 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:45:06.825318 (+ 369us) server_negotiation.cc:407] Connection header received
0926 03:45:06.825531 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:45:06.825539 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:45:06.825637 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:45:06.825751 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:45:06.827087 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:45:06.827900 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:45:06.829762 (+ 1862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:45:06.830225 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:45:06.830477 (+ 252us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:45:06.835049 (+ 4572us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.194.193:54837: BlockingRecv error: recv got EOF from 127.0.194.193:54837 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Sep 26 03:45:07 dist-test-slave-n2th krb5kdc[9020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.194.194@KRBTEST.COM for oryx/127.0.194.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:45:07 dist-test-slave-n2th krb5kdc[9020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.194.194@KRBTEST.COM for oryx/127.0.194.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:45:07.247598 10060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:45:07.237854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:36233 (local address 127.0.194.194:35231)
0926 03:45:07.238357 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0926 03:45:07.238373 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 03:45:07.238529 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:45:07.238918 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:45:07.238923 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:45:07.239376 (+ 453us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:45:07.239552 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:45:07.239560 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:45:07.240849 (+ 1289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:45:07.240855 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:45:07.241845 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:45:07.241857 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:45:07.242629 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:45:07.242634 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:45:07.242879 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:45:07.243468 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:45:07.243485 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:45:07.247287 (+ 3802us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.194.254:36233: Server oryx/127.0.194.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":262,"threads_started":1}
W20260926 03:45:07.247897 10025 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:45:07.237946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.194:35231 (local address 127.0.194.254:36233)
0926 03:45:07.238216 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0926 03:45:07.238222 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:45:07.238463 (+ 241us) server_negotiation.cc:407] Connection header received
0926 03:45:07.238681 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:45:07.238687 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:45:07.238783 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:45:07.238887 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:45:07.239716 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:45:07.240705 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:45:07.241997 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:45:07.242450 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:45:07.242584 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:45:07.247571 (+ 4987us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.194.194:35231: BlockingRecv error: recv got EOF from 127.0.194.194:35231 (error 108)
Metrics: {"server-negotiator.queue_time_us":100}
Sep 26 03:45:07 dist-test-slave-n2th krb5kdc[9020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.194.195@KRBTEST.COM for oryx/127.0.194.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:45:07 dist-test-slave-n2th krb5kdc[9020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.194.195@KRBTEST.COM for oryx/127.0.194.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:45:07.685343 10061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:45:07.675669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:36233 (local address 127.0.194.195:48827)
0926 03:45:07.676188 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0926 03:45:07.676204 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 03:45:07.676364 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:45:07.676746 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:45:07.676751 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:45:07.677224 (+ 473us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:45:07.677400 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:45:07.677411 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:45:07.678427 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:45:07.678435 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:45:07.679427 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:45:07.679439 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:45:07.680181 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:45:07.680187 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:45:07.680369 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:45:07.681018 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:45:07.681038 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:45:07.685037 (+ 3999us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.194.254:36233: Server oryx/127.0.194.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":310,"threads_started":1}
W20260926 03:45:07.685462 10025 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:45:07.675743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.195:48827 (local address 127.0.194.254:36233)
0926 03:45:07.675973 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0926 03:45:07.675980 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:45:07.676291 (+ 311us) server_negotiation.cc:407] Connection header received
0926 03:45:07.676513 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:45:07.676517 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:45:07.676576 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:45:07.676712 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:45:07.677557 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:45:07.678283 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:45:07.679572 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:45:07.680018 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:45:07.680158 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:45:07.685209 (+ 5051us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.194.195:48827: BlockingRecv error: recv got EOF from 127.0.194.195:48827 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Sep 26 03:45:07 dist-test-slave-n2th krb5kdc[9020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.194.193@KRBTEST.COM for oryx/127.0.194.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:45:07 dist-test-slave-n2th krb5kdc[9020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.194.193@KRBTEST.COM for oryx/127.0.194.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:45:07.846010 10062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:45:07.835992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:36233 (local address 127.0.194.193:50391)
0926 03:45:07.836975 (+ 983us) negotiation.cc:107] Waiting for socket to connect
0926 03:45:07.836992 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 03:45:07.837156 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:45:07.837483 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:45:07.837488 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:45:07.837941 (+ 453us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:45:07.838101 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:45:07.838109 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:45:07.839212 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:45:07.839220 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:45:07.840188 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:45:07.840199 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:45:07.840919 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:45:07.840925 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:45:07.841114 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:45:07.841838 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:45:07.841856 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:45:07.845647 (+ 3791us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.194.254:36233: Server oryx/127.0.194.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":874,"thread_start_us":771,"threads_started":1}
W20260926 03:45:07.846421 10025 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:45:07.836057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.193:50391 (local address 127.0.194.254:36233)
0926 03:45:07.836325 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0926 03:45:07.836334 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:45:07.837083 (+ 749us) server_negotiation.cc:407] Connection header received
0926 03:45:07.837275 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:45:07.837281 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:45:07.837356 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:45:07.837455 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:45:07.838237 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:45:07.839081 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:45:07.840333 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:45:07.840769 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:45:07.840903 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:45:07.846059 (+ 5156us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.194.193:50391: BlockingRecv error: recv got EOF from 127.0.194.193:50391 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Sep 26 03:45:08 dist-test-slave-n2th krb5kdc[9020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.194.194@KRBTEST.COM for oryx/127.0.194.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:45:08 dist-test-slave-n2th krb5kdc[9020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.194.194@KRBTEST.COM for oryx/127.0.194.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:45:08.258159 10063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:45:08.248867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:36233 (local address 127.0.194.194:50031)
0926 03:45:08.249393 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0926 03:45:08.249409 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 03:45:08.249530 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:45:08.249839 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:45:08.249843 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:45:08.250408 (+ 565us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:45:08.250572 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:45:08.250581 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:45:08.251628 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:45:08.251634 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:45:08.252583 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:45:08.252594 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:45:08.253528 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:45:08.253535 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:45:08.253721 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:45:08.254444 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:45:08.254471 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:45:08.257905 (+ 3434us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.194.254:36233: Server oryx/127.0.194.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":296,"threads_started":1}
W20260926 03:45:08.258352 10025 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:45:08.248964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.194:50031 (local address 127.0.194.254:36233)
0926 03:45:08.249175 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0926 03:45:08.249190 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 03:45:08.249491 (+ 301us) server_negotiation.cc:407] Connection header received
0926 03:45:08.249630 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:45:08.249634 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:45:08.249695 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:45:08.249811 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:45:08.250714 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:45:08.251492 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:45:08.252719 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:45:08.253279 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:45:08.253474 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:45:08.258122 (+ 4648us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.194.194:50031: BlockingRecv error: recv got EOF from 127.0.194.194:50031 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Sep 26 03:45:08 dist-test-slave-n2th krb5kdc[9020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.194.195@KRBTEST.COM for oryx/127.0.194.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:45:08 dist-test-slave-n2th krb5kdc[9020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.194.195@KRBTEST.COM for oryx/127.0.194.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:45:08.695914 10064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:45:08.686489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:36233 (local address 127.0.194.195:50303)
0926 03:45:08.687009 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0926 03:45:08.687024 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 03:45:08.687180 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:45:08.687530 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:45:08.687535 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:45:08.687952 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:45:08.688124 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:45:08.688132 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:45:08.689222 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:45:08.689227 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:45:08.690152 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:45:08.690163 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:45:08.690970 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:45:08.690976 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:45:08.691164 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:45:08.691821 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:45:08.691838 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:45:08.695562 (+ 3724us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.194.254:36233: Server oryx/127.0.194.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":294,"threads_started":1}
W20260926 03:45:08.696038 10025 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:45:08.686559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.195:50303 (local address 127.0.194.254:36233)
0926 03:45:08.686764 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0926 03:45:08.686771 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:45:08.687112 (+ 341us) server_negotiation.cc:407] Connection header received
0926 03:45:08.687300 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:45:08.687305 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:45:08.687375 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:45:08.687483 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:45:08.688285 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:45:08.689067 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:45:08.690293 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:45:08.690778 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:45:08.690941 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:45:08.695783 (+ 4842us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.194.195:50303: BlockingRecv error: recv got EOF from 127.0.194.195:50303 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 26 03:45:08 dist-test-slave-n2th krb5kdc[9020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.194.193@KRBTEST.COM for oryx/127.0.194.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:45:08 dist-test-slave-n2th krb5kdc[9020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.194.193@KRBTEST.COM for oryx/127.0.194.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:45:08.856420 10065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:45:08.847144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:36233 (local address 127.0.194.193:40981)
0926 03:45:08.847656 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0926 03:45:08.847673 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 03:45:08.847791 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:45:08.848070 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:45:08.848073 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:45:08.848550 (+ 477us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:45:08.848753 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:45:08.848766 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:45:08.849899 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:45:08.849910 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:45:08.850935 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:45:08.850947 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:45:08.851718 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:45:08.851723 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:45:08.851896 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:45:08.852647 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:45:08.852666 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:45:08.856051 (+ 3385us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.194.254:36233: Server oryx/127.0.194.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":235,"threads_started":1}
W20260926 03:45:08.856562 10025 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:45:08.847174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.193:40981 (local address 127.0.194.254:36233)
0926 03:45:08.847380 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0926 03:45:08.847408 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0926 03:45:08.847747 (+ 339us) server_negotiation.cc:407] Connection header received
0926 03:45:08.847875 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:45:08.847878 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:45:08.847942 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:45:08.848024 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:45:08.848926 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:45:08.849739 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:45:08.851071 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:45:08.851520 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:45:08.851622 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:45:08.856271 (+ 4649us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.194.193:40981: BlockingRecv error: recv got EOF from 127.0.194.193:40981 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 26 03:45:09 dist-test-slave-n2th krb5kdc[9020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.194.194@KRBTEST.COM for oryx/127.0.194.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:45:09 dist-test-slave-n2th krb5kdc[9020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.194.194@KRBTEST.COM for oryx/127.0.194.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:45:09.268455 10066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:45:09.259300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:36233 (local address 127.0.194.194:36537)
0926 03:45:09.259786 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0926 03:45:09.259803 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 03:45:09.259923 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:45:09.260313 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:45:09.260318 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:45:09.260734 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:45:09.260917 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:45:09.260928 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:45:09.262342 (+ 1414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:45:09.262347 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:45:09.263330 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:45:09.263341 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:45:09.264011 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:45:09.264016 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:45:09.264166 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:45:09.264756 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:45:09.264773 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:45:09.268204 (+ 3431us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.194.254:36233: Server oryx/127.0.194.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":287,"threads_started":1}
W20260926 03:45:09.268662 10025 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:45:09.259436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.194:36537 (local address 127.0.194.254:36233)
0926 03:45:09.259696 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0926 03:45:09.259702 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:45:09.259890 (+ 188us) server_negotiation.cc:407] Connection header received
0926 03:45:09.260044 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:45:09.260049 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:45:09.260128 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:45:09.260243 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:45:09.261079 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:45:09.262188 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:45:09.263462 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:45:09.263868 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:45:09.263962 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:45:09.268430 (+ 4468us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.194.194:36537: BlockingRecv error: recv got EOF from 127.0.194.194:36537 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Sep 26 03:45:09 dist-test-slave-n2th krb5kdc[9020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.194.195@KRBTEST.COM for oryx/127.0.194.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:45:09 dist-test-slave-n2th krb5kdc[9020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.194.195@KRBTEST.COM for oryx/127.0.194.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:45:09.706466 10067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:45:09.697216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:36233 (local address 127.0.194.195:46837)
0926 03:45:09.697706 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0926 03:45:09.697728 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 03:45:09.697850 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:45:09.698206 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:45:09.698210 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:45:09.698627 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:45:09.698807 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:45:09.698817 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:45:09.699831 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:45:09.699835 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:45:09.700764 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:45:09.700777 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:45:09.701436 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:45:09.701440 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:45:09.701579 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:45:09.702281 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:45:09.702299 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:45:09.706125 (+ 3826us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.194.254:36233: Server oryx/127.0.194.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":282,"threads_started":1}
W20260926 03:45:09.706566 10025 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:45:09.697321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.195:46837 (local address 127.0.194.254:36233)
0926 03:45:09.697507 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0926 03:45:09.697513 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:45:09.697835 (+ 322us) server_negotiation.cc:407] Connection header received
0926 03:45:09.697968 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:45:09.697973 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:45:09.698049 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:45:09.698169 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:45:09.698941 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:45:09.699685 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:45:09.700905 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:45:09.701305 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:45:09.701402 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:45:09.706331 (+ 4929us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.194.195:46837: BlockingRecv error: recv got EOF from 127.0.194.195:46837 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Sep 26 03:45:09 dist-test-slave-n2th krb5kdc[9020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.194.193@KRBTEST.COM for oryx/127.0.194.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:45:09 dist-test-slave-n2th krb5kdc[9020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.194.193@KRBTEST.COM for oryx/127.0.194.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:45:09.867221 10068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:45:09.858011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:36233 (local address 127.0.194.193:54599)
0926 03:45:09.858585 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0926 03:45:09.858610 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0926 03:45:09.858753 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:45:09.859066 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:45:09.859070 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:45:09.859441 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:45:09.859606 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:45:09.859614 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:45:09.860646 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:45:09.860651 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:45:09.861602 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:45:09.861614 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:45:09.862365 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:45:09.862370 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:45:09.862499 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:45:09.863164 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:45:09.863181 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:45:09.866938 (+ 3757us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.194.254:36233: Server oryx/127.0.194.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":401,"threads_started":1}
W20260926 03:45:09.867396 10025 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:45:09.858180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.193:54599 (local address 127.0.194.254:36233)
0926 03:45:09.858439 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0926 03:45:09.858444 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:45:09.858698 (+ 254us) server_negotiation.cc:407] Connection header received
0926 03:45:09.858868 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:45:09.858871 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:45:09.858932 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:45:09.859016 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:45:09.859744 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:45:09.860484 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:45:09.861737 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:45:09.862227 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:45:09.862324 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:45:09.867151 (+ 4827us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.194.193:54599: BlockingRecv error: recv got EOF from 127.0.194.193:54599 (error 108)
Metrics: {"server-negotiator.queue_time_us":114}
Sep 26 03:45:10 dist-test-slave-n2th krb5kdc[9020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.194.194@KRBTEST.COM for oryx/127.0.194.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:45:10 dist-test-slave-n2th krb5kdc[9020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.194.194@KRBTEST.COM for oryx/127.0.194.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:45:10.279327 10069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:45:10.269755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:36233 (local address 127.0.194.194:56819)
0926 03:45:10.270441 (+ 686us) negotiation.cc:107] Waiting for socket to connect
0926 03:45:10.270466 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0926 03:45:10.270623 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:45:10.271035 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:45:10.271040 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:45:10.271432 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:45:10.271685 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:45:10.271697 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:45:10.272880 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:45:10.272885 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:45:10.273817 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:45:10.273828 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:45:10.274487 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:45:10.274491 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:45:10.274637 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:45:10.275336 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:45:10.275353 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:45:10.278947 (+ 3594us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.194.254:36233: Server oryx/127.0.194.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":317,"threads_started":1}
W20260926 03:45:10.279459 10025 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:45:10.269841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.194:56819 (local address 127.0.194.254:36233)
0926 03:45:10.270213 (+ 372us) server_negotiation.cc:206] Beginning negotiation
0926 03:45:10.270219 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:45:10.270580 (+ 361us) server_negotiation.cc:407] Connection header received
0926 03:45:10.270817 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:45:10.270821 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:45:10.270887 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:45:10.271002 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:45:10.271860 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:45:10.272725 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:45:10.273944 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:45:10.274361 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:45:10.274456 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:45:10.279217 (+ 4761us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.194.194:56819: BlockingRecv error: recv got EOF from 127.0.194.194:56819 (error 108)
Metrics: {"server-negotiator.queue_time_us":140}
I20260926 03:45:10.319231 779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu pid 9607
I20260926 03:45:10.332355 10071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:45:10.320498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49022 (local address 127.0.194.193:41821)
0926 03:45:10.321157 (+ 659us) server_negotiation.cc:206] Beginning negotiation
0926 03:45:10.321164 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:45:10.321189 (+ 25us) server_negotiation.cc:407] Connection header received
0926 03:45:10.321251 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:45:10.321257 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:45:10.321486 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:45:10.321660 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:45:10.323311 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:45:10.324437 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:45:10.325686 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:45:10.326241 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:45:10.326342 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:45:10.328943 (+ 2601us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:45:10.328952 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:45:10.328966 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:45:10.329014 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:45:10.331831 (+ 2817us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.194.193@KRBTEST.COM not found in keytab (ticket kvno 2)
0926 03:45:10.332109 (+ 278us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:49022: Request ticket server kudu/127.0.194.193@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":494,"thread_start_us":358,"threads_started":1}
W20260926 03:45:10.332477 10071 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:49022: Request ticket server kudu/127.0.194.193@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260926 03:45:10.332510 779 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu pid 9607: Not authorized: Client connection negotiation failed: client connection to 127.0.194.193:41821: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.194.193@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260926 03:45:10.332755 779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu with pid 9607
I20260926 03:45:10.350584 779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu pid 9759
W20260926 03:45:10.363278 779 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu pid 9759: Not authorized: Client connection negotiation failed: client connection to 127.0.194.194:42665: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.194.194@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260926 03:45:10.363384 10072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:45:10.351694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53444 (local address 127.0.194.194:42665)
0926 03:45:10.352342 (+ 648us) server_negotiation.cc:206] Beginning negotiation
0926 03:45:10.352351 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:45:10.352385 (+ 34us) server_negotiation.cc:407] Connection header received
0926 03:45:10.352458 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:45:10.352466 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:45:10.352808 (+ 342us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:45:10.353038 (+ 230us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:45:10.354249 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:45:10.355505 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:45:10.356667 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:45:10.357238 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:45:10.357334 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:45:10.359847 (+ 2513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:45:10.359855 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:45:10.359869 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:45:10.359908 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:45:10.362504 (+ 2596us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.194.194@KRBTEST.COM not found in keytab (ticket kvno 2)
0926 03:45:10.362999 (+ 495us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:53444: Request ticket server kudu/127.0.194.194@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":439,"thread_start_us":290,"threads_started":1}
I20260926 03:45:10.363451 779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu with pid 9759
W20260926 03:45:10.363508 10072 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:53444: Request ticket server kudu/127.0.194.194@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260926 03:45:10.379997 779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu pid 9892
I20260926 03:45:10.392352 10073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:45:10.381130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33152 (local address 127.0.194.195:45223)
0926 03:45:10.381755 (+ 625us) server_negotiation.cc:206] Beginning negotiation
0926 03:45:10.381763 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:45:10.381788 (+ 25us) server_negotiation.cc:407] Connection header received
0926 03:45:10.381853 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:45:10.381860 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:45:10.382097 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:45:10.382312 (+ 215us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:45:10.383754 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:45:10.384756 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:45:10.385981 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:45:10.386549 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:45:10.386706 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:45:10.389189 (+ 2483us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:45:10.389197 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:45:10.389211 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:45:10.389258 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:45:10.391666 (+ 2408us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.194.195@KRBTEST.COM not found in keytab (ticket kvno 2)
0926 03:45:10.392010 (+ 344us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:33152: Request ticket server kudu/127.0.194.195@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":446,"thread_start_us":330,"threads_started":1}
W20260926 03:45:10.392475 10073 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:33152: Request ticket server kudu/127.0.194.195@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260926 03:45:10.392465 779 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu pid 9892: Not authorized: Client connection negotiation failed: client connection to 127.0.194.195:45223: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.194.195@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260926 03:45:10.392661 779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu with pid 9892
I20260926 03:45:10.409264 779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu pid 9538
I20260926 03:45:10.410599 9600 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:45:10.523793 779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu with pid 9538
2026-09-26T03:45:10Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (21299 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-26T03:45:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:45:10Z Disabled control of system clock
I20260926 03:45:10.602005 779 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
/tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790394221564982-779-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790394221564982-779-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790394221564982-779-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790394221564982-779-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.194.254:38077
--webserver_interface=127.0.194.254
--webserver_port=0
--builtin_ntp_servers=127.0.194.212:34085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.194.254:38077
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260926 03:45:10.937502 10078 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:45:10.937825 10078 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:45:10.937903 10078 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:45:10.945307 10078 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:45:10.945423 10078 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:45:10.945461 10078 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:45:10.945520 10078 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:45:10.945564 10078 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:45:10.954978 10078 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.194.212:34085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790394221564982-779-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790394221564982-779-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.194.254:38077
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.194.254:38077
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790394221564982-779-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.194.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790394221564982-779-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:45:10.956245 10078 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:45:10.958014 10078 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:45:10.970196 10083 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:45:10.970234 10084 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:45:10.970402 10086 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:45:10.971390 10078 server_base.cc:1034] running on GCE node
I20260926 03:45:10.972282 10078 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:45:10.973804 10078 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:45:10.975030 10078 hybrid_clock.cc:648] HybridClock initialized: now 1790394310974979 us; error 48 us; skew 500 ppm
I20260926 03:45:10.975394 10078 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 03:45:10.977607 10078 webserver.cc:460] Webserver started at http://127.0.194.254:43529/ using document root <none> and password file <none>
I20260926 03:45:10.978317 10078 fs_manager.cc:359] Metadata directory not provided
I20260926 03:45:10.978444 10078 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:45:10.978833 10078 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:45:10.981774 10078 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790394221564982-779-0/minicluster-data/master-0/data/instance:
uuid: "a16c7504769f40e39547a51344a86614"
format_stamp: "Formatted at 2026-09-26 03:45:10 on dist-test-slave-n2th"
server_key: "c7a1e2dc056ceff8b826cb926b8b09b5"
server_key_iv: "99e4d443c4cea07ead9c627c201225aa"
server_key_version: "encryptionkey@0"
I20260926 03:45:10.982590 10078 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790394221564982-779-0/minicluster-data/master-0/wal/instance:
uuid: "a16c7504769f40e39547a51344a86614"
format_stamp: "Formatted at 2026-09-26 03:45:10 on dist-test-slave-n2th"
server_key: "c7a1e2dc056ceff8b826cb926b8b09b5"
server_key_iv: "99e4d443c4cea07ead9c627c201225aa"
server_key_version: "encryptionkey@0"
I20260926 03:45:10.988900 10078 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260926 03:45:10.992606 10092 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:45:10.994196 10078 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 03:45:10.994407 10078 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790394221564982-779-0/minicluster-data/master-0/data,/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790394221564982-779-0/minicluster-data/master-0/wal
uuid: "a16c7504769f40e39547a51344a86614"
format_stamp: "Formatted at 2026-09-26 03:45:10 on dist-test-slave-n2th"
server_key: "c7a1e2dc056ceff8b826cb926b8b09b5"
server_key_iv: "99e4d443c4cea07ead9c627c201225aa"
server_key_version: "encryptionkey@0"
I20260926 03:45:10.994658 10078 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790394221564982-779-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790394221564982-779-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790394221564982-779-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:45:11.010232 10078 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:45:11.011135 10078 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:45:11.011380 10078 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:45:11.030246 10078 rpc_server.cc:307] RPC server started. Bound to: 127.0.194.254:38077
I20260926 03:45:11.030269 10143 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.194.254:38077 every 8 connection(s)
I20260926 03:45:11.031759 10078 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790394221564982-779-0/minicluster-data/master-0/data/info.pb
I20260926 03:45:11.035549 779 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu as pid 10078
I20260926 03:45:11.035755 779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790394221564982-779-0/minicluster-data/master-0/wal/instance
I20260926 03:45:11.035980 10144 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:45:11.036494 779 external_mini_cluster.cc:1428] Setting key ed8bc8f62f46c5d2920ce1b841a1239f
I20260926 03:45:11.049662 10144 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a16c7504769f40e39547a51344a86614: Bootstrap starting.
I20260926 03:45:11.051641 10147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:45:11.038448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35940 (local address 127.0.194.254:38077)
0926 03:45:11.039530 (+ 1082us) server_negotiation.cc:206] Beginning negotiation
0926 03:45:11.039541 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 03:45:11.039586 (+ 45us) server_negotiation.cc:407] Connection header received
0926 03:45:11.041835 (+ 2249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:45:11.041857 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:45:11.042403 (+ 546us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:45:11.042830 (+ 427us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:45:11.043406 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:45:11.044690 (+ 1284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:45:11.046697 (+ 2007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:45:11.047584 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:45:11.047732 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:45:11.049130 (+ 1398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:45:11.049137 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:45:11.049154 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:45:11.049723 (+ 569us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:45:11.049842 (+ 119us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:45:11.049875 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:45:11.049996 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0926 03:45:11.050229 (+ 233us) server_negotiation.cc:299] Negotiation successful
0926 03:45:11.050598 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":651,"thread_start_us":426,"threads_started":1}
I20260926 03:45:11.053462 10144 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a16c7504769f40e39547a51344a86614: Neither blocks nor log segments found. Creating new log.
I20260926 03:45:11.054605 10144 log.cc:824] T 00000000000000000000000000000000 P a16c7504769f40e39547a51344a86614: Log is configured to *not* fsync() on all Append() calls
I20260926 03:45:11.057466 10144 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a16c7504769f40e39547a51344a86614: No bootstrap required, opened a new log
I20260926 03:45:11.065335 10144 raft_consensus.cc:348] T 00000000000000000000000000000000 P a16c7504769f40e39547a51344a86614 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a16c7504769f40e39547a51344a86614" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 38077 } }
I20260926 03:45:11.065747 10144 raft_consensus.cc:374] T 00000000000000000000000000000000 P a16c7504769f40e39547a51344a86614 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:45:11.065853 10144 raft_consensus.cc:729] T 00000000000000000000000000000000 P a16c7504769f40e39547a51344a86614 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a16c7504769f40e39547a51344a86614, State: Initialized, Role: FOLLOWER
I20260926 03:45:11.066557 10144 consensus_queue.cc:260] T 00000000000000000000000000000000 P a16c7504769f40e39547a51344a86614 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a16c7504769f40e39547a51344a86614" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 38077 } }
I20260926 03:45:11.066855 10144 raft_consensus.cc:388] T 00000000000000000000000000000000 P a16c7504769f40e39547a51344a86614 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:45:11.066992 10144 raft_consensus.cc:482] T 00000000000000000000000000000000 P a16c7504769f40e39547a51344a86614 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:45:11.067170 10144 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a16c7504769f40e39547a51344a86614 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:45:11.069675 10144 raft_consensus.cc:504] T 00000000000000000000000000000000 P a16c7504769f40e39547a51344a86614 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a16c7504769f40e39547a51344a86614" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 38077 } }
I20260926 03:45:11.070252 10144 leader_election.cc:302] T 00000000000000000000000000000000 P a16c7504769f40e39547a51344a86614 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a16c7504769f40e39547a51344a86614; no voters:
I20260926 03:45:11.071007 10144 leader_election.cc:288] T 00000000000000000000000000000000 P a16c7504769f40e39547a51344a86614 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:45:11.071177 10149 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a16c7504769f40e39547a51344a86614 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:45:11.071907 10149 raft_consensus.cc:686] T 00000000000000000000000000000000 P a16c7504769f40e39547a51344a86614 [term 1 LEADER]: Becoming Leader. State: Replica: a16c7504769f40e39547a51344a86614, State: Running, Role: LEADER
I20260926 03:45:11.072415 10149 consensus_queue.cc:237] T 00000000000000000000000000000000 P a16c7504769f40e39547a51344a86614 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a16c7504769f40e39547a51344a86614" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 38077 } }
I20260926 03:45:11.073160 10144 sys_catalog.cc:564] T 00000000000000000000000000000000 P a16c7504769f40e39547a51344a86614 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:45:11.075188 10151 sys_catalog.cc:455] T 00000000000000000000000000000000 P a16c7504769f40e39547a51344a86614 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a16c7504769f40e39547a51344a86614. Latest consensus state: current_term: 1 leader_uuid: "a16c7504769f40e39547a51344a86614" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a16c7504769f40e39547a51344a86614" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 38077 } } }
I20260926 03:45:11.075634 10151 sys_catalog.cc:458] T 00000000000000000000000000000000 P a16c7504769f40e39547a51344a86614 [sys.catalog]: This master's current role is: LEADER
I20260926 03:45:11.075575 10150 sys_catalog.cc:455] T 00000000000000000000000000000000 P a16c7504769f40e39547a51344a86614 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a16c7504769f40e39547a51344a86614" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a16c7504769f40e39547a51344a86614" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 38077 } } }
I20260926 03:45:11.075883 10150 sys_catalog.cc:458] T 00000000000000000000000000000000 P a16c7504769f40e39547a51344a86614 [sys.catalog]: This master's current role is: LEADER
I20260926 03:45:11.078015 10157 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:45:11.084106 10157 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:45:11.092965 10157 catalog_manager.cc:1347] Generated new cluster ID: f59d10244e5543ec86e58d077ac22099
I20260926 03:45:11.093101 10157 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:45:11.111263 10157 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:45:11.112586 10157 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:45:11.122604 10157 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a16c7504769f40e39547a51344a86614: Generated new TSK 0
I20260926 03:45:11.123822 10157 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 03:45:11.139753 779 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
/tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790394221564982-779-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790394221564982-779-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790394221564982-779-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790394221564982-779-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.194.193:0
--local_ip_for_outbound_sockets=127.0.194.193
--webserver_interface=127.0.194.193
--webserver_port=0
--tserver_master_addrs=127.0.194.254:38077
--builtin_ntp_servers=127.0.194.212:34085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260926 03:45:11.481666 10168 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:45:11.482009 10168 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:45:11.482182 10168 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:45:11.489794 10168 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:45:11.489962 10168 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:45:11.490157 10168 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.194.193
I20260926 03:45:11.500191 10168 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.194.212:34085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790394221564982-779-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790394221564982-779-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.194.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790394221564982-779-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.194.193
--webserver_port=0
--tserver_master_addrs=127.0.194.254:38077
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.194.193
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790394221564982-779-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:45:11.501546 10168 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:45:11.503535 10168 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:45:11.517473 10176 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:45:11.517441 10173 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:45:11.517539 10174 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:45:11.520143 10168 server_base.cc:1034] running on GCE node
I20260926 03:45:11.520813 10168 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:45:11.521806 10168 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:45:11.523042 10168 hybrid_clock.cc:648] HybridClock initialized: now 1790394311523000 us; error 56 us; skew 500 ppm
I20260926 03:45:11.523383 10168 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 03:45:11.529341 10168 webserver.cc:460] Webserver started at http://127.0.194.193:46661/ using document root <none> and password file <none>
I20260926 03:45:11.530071 10168 fs_manager.cc:359] Metadata directory not provided
I20260926 03:45:11.530210 10168 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:45:11.530551 10168 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:45:11.533484 10168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790394221564982-779-0/minicluster-data/ts-0/data/instance:
uuid: "80959d6d824241b7966e2cdd5864ef8a"
format_stamp: "Formatted at 2026-09-26 03:45:11 on dist-test-slave-n2th"
server_key: "542d9bdfcefb9f099b97fcbe0931bbe5"
server_key_iv: "fcdefad50577938d4a2020b242695822"
server_key_version: "encryptionkey@0"
I20260926 03:45:11.534305 10168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790394221564982-779-0/minicluster-data/ts-0/wal/instance:
uuid: "80959d6d824241b7966e2cdd5864ef8a"
format_stamp: "Formatted at 2026-09-26 03:45:11 on dist-test-slave-n2th"
server_key: "542d9bdfcefb9f099b97fcbe0931bbe5"
server_key_iv: "fcdefad50577938d4a2020b242695822"
server_key_version: "encryptionkey@0"
I20260926 03:45:11.539551 10168 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260926 03:45:11.542923 10182 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:45:11.544200 10168 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 03:45:11.544384 10168 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790394221564982-779-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790394221564982-779-0/minicluster-data/ts-0/wal
uuid: "80959d6d824241b7966e2cdd5864ef8a"
format_stamp: "Formatted at 2026-09-26 03:45:11 on dist-test-slave-n2th"
server_key: "542d9bdfcefb9f099b97fcbe0931bbe5"
server_key_iv: "fcdefad50577938d4a2020b242695822"
server_key_version: "encryptionkey@0"
I20260926 03:45:11.544600 10168 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790394221564982-779-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790394221564982-779-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790394221564982-779-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:45:11.564355 10168 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:45:11.565397 10168 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:45:11.565703 10168 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:45:11.566679 10168 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:45:11.568140 10168 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:45:11.568280 10168 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:45:11.568424 10168 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:45:11.568519 10168 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:45:11.602818 10168 rpc_server.cc:307] RPC server started. Bound to: 127.0.194.193:39775
I20260926 03:45:11.602908 10294 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.194.193:39775 every 8 connection(s)
I20260926 03:45:11.604204 10168 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790394221564982-779-0/minicluster-data/ts-0/data/info.pb
I20260926 03:45:11.605558 779 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu as pid 10168
I20260926 03:45:11.605760 779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790394221564982-779-0/minicluster-data/ts-0/wal/instance
I20260926 03:45:11.606575 779 external_mini_cluster.cc:1428] Setting key 7e07b1f5e4d1b523b1bdd694231b91cf
I20260926 03:45:11.609566 779 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
/tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790394221564982-779-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790394221564982-779-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790394221564982-779-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790394221564982-779-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.194.194:0
--local_ip_for_outbound_sockets=127.0.194.194
--webserver_interface=127.0.194.194
--webserver_port=0
--tserver_master_addrs=127.0.194.254:38077
--builtin_ntp_servers=127.0.194.212:34085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260926 03:45:11.617530 10297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:45:11.607168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.193:60539 (local address 127.0.194.254:38077)
0926 03:45:11.607630 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0926 03:45:11.607634 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:45:11.609075 (+ 1441us) server_negotiation.cc:407] Connection header received
0926 03:45:11.610929 (+ 1854us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:45:11.610934 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:45:11.611028 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:45:11.611153 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:45:11.612250 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:45:11.612992 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:45:11.614586 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:45:11.615052 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:45:11.615206 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:45:11.616665 (+ 1459us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:45:11.616670 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:45:11.616673 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:45:11.616692 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:45:11.616738 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:45:11.616750 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:45:11.616858 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0926 03:45:11.617087 (+ 229us) server_negotiation.cc:299] Negotiation successful
0926 03:45:11.617187 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":367,"thread_start_us":247,"threads_started":1}
I20260926 03:45:11.618608 10298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:45:11.607890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:38077 (local address 127.0.194.193:60539)
0926 03:45:11.608935 (+ 1045us) negotiation.cc:107] Waiting for socket to connect
0926 03:45:11.608972 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 03:45:11.610611 (+ 1639us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:45:11.611392 (+ 781us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:45:11.611401 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:45:11.611496 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:45:11.612085 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:45:11.612101 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:45:11.613151 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:45:11.613161 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:45:11.614418 (+ 1257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:45:11.614434 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:45:11.615312 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:45:11.615318 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:45:11.615539 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:45:11.616302 (+ 763us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 03:45:11.616305 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:45:11.616361 (+ 56us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 03:45:11.616456 (+ 95us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:45:11.616857 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:45:11.616870 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:45:11.616880 (+ 10us) client_negotiation.cc:769] Sending connection context
0926 03:45:11.617080 (+ 200us) client_negotiation.cc:240] Negotiation successful
0926 03:45:11.617503 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":532,"thread_start_us":311,"threads_started":1}
I20260926 03:45:11.620293 10295 heartbeater.cc:347] Connected to a master server at 127.0.194.254:38077
I20260926 03:45:11.620613 10295 heartbeater.cc:464] Registering TS with master...
I20260926 03:45:11.621389 10295 heartbeater.cc:511] Master 127.0.194.254:38077 requested a full tablet report, sending...
I20260926 03:45:11.623822 10109 ts_manager.cc:194] Registered new tserver with Master: 80959d6d824241b7966e2cdd5864ef8a (127.0.194.193:39775)
I20260926 03:45:11.626101 10109 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.194.193:60539
W20260926 03:45:11.944486 10299 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:45:11.944808 10299 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:45:11.944896 10299 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:45:11.952080 10299 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:45:11.952214 10299 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:45:11.952327 10299 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.194.194
I20260926 03:45:11.961429 10299 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.194.212:34085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790394221564982-779-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790394221564982-779-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.194.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790394221564982-779-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.194.194
--webserver_port=0
--tserver_master_addrs=127.0.194.254:38077
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.194.194
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790394221564982-779-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:45:11.962801 10299 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:45:11.964649 10299 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:45:11.978685 10305 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:45:11.978965 10304 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:45:11.978868 10307 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:45:11.979473 10299 server_base.cc:1034] running on GCE node
I20260926 03:45:11.980055 10299 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:45:11.981009 10299 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:45:11.982224 10299 hybrid_clock.cc:648] HybridClock initialized: now 1790394311982154 us; error 56 us; skew 500 ppm
I20260926 03:45:11.982565 10299 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 03:45:11.985001 10299 webserver.cc:460] Webserver started at http://127.0.194.194:36173/ using document root <none> and password file <none>
I20260926 03:45:11.985759 10299 fs_manager.cc:359] Metadata directory not provided
I20260926 03:45:11.985883 10299 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:45:11.986241 10299 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:45:11.989264 10299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790394221564982-779-0/minicluster-data/ts-1/data/instance:
uuid: "f79689e1e41a4eeab94de27322d2c7c4"
format_stamp: "Formatted at 2026-09-26 03:45:11 on dist-test-slave-n2th"
server_key: "16fd8ccc0210361ad81f8080dd358c86"
server_key_iv: "5570e735845e3f3c4173f2c6ece7bfbd"
server_key_version: "encryptionkey@0"
I20260926 03:45:11.990437 10299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790394221564982-779-0/minicluster-data/ts-1/wal/instance:
uuid: "f79689e1e41a4eeab94de27322d2c7c4"
format_stamp: "Formatted at 2026-09-26 03:45:11 on dist-test-slave-n2th"
server_key: "16fd8ccc0210361ad81f8080dd358c86"
server_key_iv: "5570e735845e3f3c4173f2c6ece7bfbd"
server_key_version: "encryptionkey@0"
I20260926 03:45:11.996479 10299 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.000s sys 0.005s
I20260926 03:45:12.000326 10313 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:45:12.001634 10299 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 03:45:12.001824 10299 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790394221564982-779-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790394221564982-779-0/minicluster-data/ts-1/wal
uuid: "f79689e1e41a4eeab94de27322d2c7c4"
format_stamp: "Formatted at 2026-09-26 03:45:11 on dist-test-slave-n2th"
server_key: "16fd8ccc0210361ad81f8080dd358c86"
server_key_iv: "5570e735845e3f3c4173f2c6ece7bfbd"
server_key_version: "encryptionkey@0"
I20260926 03:45:12.002054 10299 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790394221564982-779-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790394221564982-779-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790394221564982-779-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:45:12.020140 10299 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:45:12.021085 10299 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:45:12.021400 10299 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:45:12.022395 10299 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:45:12.023893 10299 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:45:12.024014 10299 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:45:12.024154 10299 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:45:12.024248 10299 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:45:12.059453 10299 rpc_server.cc:307] RPC server started. Bound to: 127.0.194.194:36209
I20260926 03:45:12.059507 10425 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.194.194:36209 every 8 connection(s)
I20260926 03:45:12.060812 10299 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790394221564982-779-0/minicluster-data/ts-1/data/info.pb
I20260926 03:45:12.064117 779 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu as pid 10299
I20260926 03:45:12.064324 779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790394221564982-779-0/minicluster-data/ts-1/wal/instance
I20260926 03:45:12.064927 779 external_mini_cluster.cc:1428] Setting key 3cd7a6e6283a1c30f235aaaaf71fa6ac
I20260926 03:45:12.067977 779 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
/tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790394221564982-779-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790394221564982-779-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790394221564982-779-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790394221564982-779-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.194.195:0
--local_ip_for_outbound_sockets=127.0.194.195
--webserver_interface=127.0.194.195
--webserver_port=0
--tserver_master_addrs=127.0.194.254:38077
--builtin_ntp_servers=127.0.194.212:34085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260926 03:45:12.074304 10297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:45:12.063417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.194:51141 (local address 127.0.194.254:38077)
0926 03:45:12.063605 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0926 03:45:12.063610 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:45:12.065445 (+ 1835us) server_negotiation.cc:407] Connection header received
0926 03:45:12.067556 (+ 2111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:45:12.067561 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:45:12.067645 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:45:12.067768 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:45:12.068991 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:45:12.069745 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:45:12.071470 (+ 1725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:45:12.071864 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:45:12.072002 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:45:12.073422 (+ 1420us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:45:12.073428 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:45:12.073432 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:45:12.073461 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:45:12.073493 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:45:12.073505 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:45:12.073603 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 03:45:12.073880 (+ 277us) server_negotiation.cc:299] Negotiation successful
0926 03:45:12.073983 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 03:45:12.075493 10428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:45:12.064218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:38077 (local address 127.0.194.194:51141)
0926 03:45:12.065275 (+ 1057us) negotiation.cc:107] Waiting for socket to connect
0926 03:45:12.065320 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 03:45:12.067287 (+ 1967us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:45:12.068022 (+ 735us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:45:12.068032 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:45:12.068190 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:45:12.068821 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:45:12.068839 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:45:12.069913 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:45:12.069918 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:45:12.071313 (+ 1395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:45:12.071330 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:45:12.072095 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:45:12.072102 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:45:12.072342 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:45:12.073095 (+ 753us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 03:45:12.073097 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:45:12.073139 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 03:45:12.073219 (+ 80us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:45:12.073601 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:45:12.073613 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:45:12.073624 (+ 11us) client_negotiation.cc:769] Sending connection context
0926 03:45:12.073873 (+ 249us) client_negotiation.cc:240] Negotiation successful
0926 03:45:12.074302 (+ 429us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":306,"threads_started":1}
I20260926 03:45:12.076936 10426 heartbeater.cc:347] Connected to a master server at 127.0.194.254:38077
I20260926 03:45:12.077273 10426 heartbeater.cc:464] Registering TS with master...
I20260926 03:45:12.078128 10426 heartbeater.cc:511] Master 127.0.194.254:38077 requested a full tablet report, sending...
I20260926 03:45:12.079743 10109 ts_manager.cc:194] Registered new tserver with Master: f79689e1e41a4eeab94de27322d2c7c4 (127.0.194.194:36209)
I20260926 03:45:12.081013 10109 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.194.194:51141
W20260926 03:45:12.404999 10429 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:45:12.405385 10429 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:45:12.405499 10429 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:45:12.412909 10429 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:45:12.413014 10429 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:45:12.413173 10429 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.194.195
I20260926 03:45:12.423350 10429 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.194.212:34085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790394221564982-779-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790394221564982-779-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.194.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790394221564982-779-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.194.195
--webserver_port=0
--tserver_master_addrs=127.0.194.254:38077
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.194.195
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790394221564982-779-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:45:12.424693 10429 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:45:12.426508 10429 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:45:12.440145 10434 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:45:12.440233 10435 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:45:12.440183 10437 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:45:12.444121 10429 server_base.cc:1034] running on GCE node
I20260926 03:45:12.444851 10429 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:45:12.445894 10429 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:45:12.447129 10429 hybrid_clock.cc:648] HybridClock initialized: now 1790394312447067 us; error 63 us; skew 500 ppm
I20260926 03:45:12.447465 10429 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 03:45:12.449878 10429 webserver.cc:460] Webserver started at http://127.0.194.195:36539/ using document root <none> and password file <none>
I20260926 03:45:12.450613 10429 fs_manager.cc:359] Metadata directory not provided
I20260926 03:45:12.450779 10429 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:45:12.451122 10429 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:45:12.453987 10429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790394221564982-779-0/minicluster-data/ts-2/data/instance:
uuid: "1b2a7019ac4b40b6851e94b875a3f83a"
format_stamp: "Formatted at 2026-09-26 03:45:12 on dist-test-slave-n2th"
server_key: "035950325160a219da18fe0358f6463d"
server_key_iv: "7ff974d6e860a8c921c41623c6107b1b"
server_key_version: "encryptionkey@0"
I20260926 03:45:12.454859 10429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790394221564982-779-0/minicluster-data/ts-2/wal/instance:
uuid: "1b2a7019ac4b40b6851e94b875a3f83a"
format_stamp: "Formatted at 2026-09-26 03:45:12 on dist-test-slave-n2th"
server_key: "035950325160a219da18fe0358f6463d"
server_key_iv: "7ff974d6e860a8c921c41623c6107b1b"
server_key_version: "encryptionkey@0"
I20260926 03:45:12.460183 10429 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 03:45:12.463714 10443 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:45:12.465080 10429 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 03:45:12.465261 10429 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790394221564982-779-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790394221564982-779-0/minicluster-data/ts-2/wal
uuid: "1b2a7019ac4b40b6851e94b875a3f83a"
format_stamp: "Formatted at 2026-09-26 03:45:12 on dist-test-slave-n2th"
server_key: "035950325160a219da18fe0358f6463d"
server_key_iv: "7ff974d6e860a8c921c41623c6107b1b"
server_key_version: "encryptionkey@0"
I20260926 03:45:12.465502 10429 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790394221564982-779-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790394221564982-779-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790394221564982-779-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:45:12.488096 10429 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:45:12.489006 10429 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:45:12.489328 10429 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:45:12.490341 10429 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:45:12.491851 10429 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:45:12.491978 10429 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:45:12.492116 10429 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:45:12.492211 10429 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:45:12.525445 10429 rpc_server.cc:307] RPC server started. Bound to: 127.0.194.195:43985
I20260926 03:45:12.525540 10555 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.194.195:43985 every 8 connection(s)
I20260926 03:45:12.526827 10429 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790394221564982-779-0/minicluster-data/ts-2/data/info.pb
I20260926 03:45:12.532624 779 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu as pid 10429
I20260926 03:45:12.532784 779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790394221564982-779-0/minicluster-data/ts-2/wal/instance
I20260926 03:45:12.533336 779 external_mini_cluster.cc:1428] Setting key 29737a187b4a8833f032d42972dc6c17
I20260926 03:45:12.540689 10297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:45:12.529526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.195:41197 (local address 127.0.194.254:38077)
0926 03:45:12.529735 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0926 03:45:12.529742 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:45:12.531301 (+ 1559us) server_negotiation.cc:407] Connection header received
0926 03:45:12.533502 (+ 2201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:45:12.533507 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:45:12.533579 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:45:12.533669 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:45:12.535089 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:45:12.535941 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:45:12.537880 (+ 1939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:45:12.538323 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:45:12.538456 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:45:12.539871 (+ 1415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:45:12.539876 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:45:12.539879 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:45:12.539896 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:45:12.539925 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:45:12.539937 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:45:12.540031 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0926 03:45:12.540259 (+ 228us) server_negotiation.cc:299] Negotiation successful
0926 03:45:12.540349 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260926 03:45:12.541636 10558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:45:12.530052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:38077 (local address 127.0.194.195:41197)
0926 03:45:12.531070 (+ 1018us) negotiation.cc:107] Waiting for socket to connect
0926 03:45:12.531133 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0926 03:45:12.533149 (+ 2016us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:45:12.533965 (+ 816us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:45:12.533974 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:45:12.534065 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:45:12.534921 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:45:12.534945 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:45:12.536100 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:45:12.536105 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:45:12.537691 (+ 1586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:45:12.537703 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:45:12.538550 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:45:12.538565 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:45:12.538854 (+ 289us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:45:12.539553 (+ 699us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 03:45:12.539555 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:45:12.539594 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 03:45:12.539677 (+ 83us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:45:12.540031 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:45:12.540043 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:45:12.540053 (+ 10us) client_negotiation.cc:769] Sending connection context
0926 03:45:12.540236 (+ 183us) client_negotiation.cc:240] Negotiation successful
0926 03:45:12.540626 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":277,"threads_started":1}
I20260926 03:45:12.542974 10556 heartbeater.cc:347] Connected to a master server at 127.0.194.254:38077
I20260926 03:45:12.543278 10556 heartbeater.cc:464] Registering TS with master...
I20260926 03:45:12.544027 10556 heartbeater.cc:511] Master 127.0.194.254:38077 requested a full tablet report, sending...
I20260926 03:45:12.545352 10109 ts_manager.cc:194] Registered new tserver with Master: 1b2a7019ac4b40b6851e94b875a3f83a (127.0.194.195:43985)
I20260926 03:45:12.546099 10109 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.194.195:41197
I20260926 03:45:12.548904 779 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260926 03:45:12.555356 10297 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:45:12.553544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35950 (local address 127.0.194.254:38077)
0926 03:45:12.553827 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0926 03:45:12.553836 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:45:12.554170 (+ 334us) server_negotiation.cc:407] Connection header received
0926 03:45:12.554388 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:45:12.554393 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:45:12.554477 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:45:12.554584 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:45:12.555102 (+ 518us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35950: BlockingRecv error: recv got EOF from 127.0.0.1:35950 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 03:45:12.556319 779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu pid 10168
I20260926 03:45:12.564289 10565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:45:12.557238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38760 (local address 127.0.194.193:39775)
0926 03:45:12.557774 (+ 536us) server_negotiation.cc:206] Beginning negotiation
0926 03:45:12.557782 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:45:12.557816 (+ 34us) server_negotiation.cc:407] Connection header received
0926 03:45:12.557875 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:45:12.557881 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:45:12.558085 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:45:12.558254 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:45:12.558660 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:45:12.559670 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:45:12.560920 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:45:12.561493 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:45:12.561624 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:45:12.562776 (+ 1152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:45:12.562788 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:45:12.562804 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:45:12.563388 (+ 584us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:45:12.563435 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:45:12.563458 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:45:12.563583 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0926 03:45:12.563799 (+ 216us) server_negotiation.cc:299] Negotiation successful
0926 03:45:12.563944 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":226,"threads_started":1}
I20260926 03:45:12.565596 10290 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:45:12.647409 10295 heartbeater.cc:503] Master 127.0.194.254:38077 was elected leader, sending a full tablet report...
I20260926 03:45:12.648597 779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu with pid 10168
W20260926 03:45:12.665897 10096 connection.cc:535] server connection from 127.0.194.193:60539 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 03:45:12.666261 779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu pid 10299
I20260926 03:45:12.674294 10567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:45:12.667238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33804 (local address 127.0.194.194:36209)
0926 03:45:12.667774 (+ 536us) server_negotiation.cc:206] Beginning negotiation
0926 03:45:12.667782 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:45:12.667810 (+ 28us) server_negotiation.cc:407] Connection header received
0926 03:45:12.667872 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:45:12.667879 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:45:12.668090 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:45:12.668298 (+ 208us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:45:12.668686 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:45:12.669789 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:45:12.671070 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:45:12.671636 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:45:12.671740 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:45:12.672751 (+ 1011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:45:12.672758 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:45:12.672773 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:45:12.673485 (+ 712us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:45:12.673530 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:45:12.673549 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:45:12.673634 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0926 03:45:12.673789 (+ 155us) server_negotiation.cc:299] Negotiation successful
0926 03:45:12.673951 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":395,"thread_start_us":278,"threads_started":1}
I20260926 03:45:12.675670 10421 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:45:12.756974 779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu with pid 10299
I20260926 03:45:12.772811 779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu pid 10429
I20260926 03:45:12.780805 10569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:45:12.773793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56194 (local address 127.0.194.195:43985)
0926 03:45:12.774372 (+ 579us) server_negotiation.cc:206] Beginning negotiation
0926 03:45:12.774379 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:45:12.774407 (+ 28us) server_negotiation.cc:407] Connection header received
0926 03:45:12.774472 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:45:12.774478 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:45:12.774798 (+ 320us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:45:12.774925 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:45:12.775334 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:45:12.776347 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:45:12.777551 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:45:12.778144 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:45:12.778280 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:45:12.779402 (+ 1122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:45:12.779411 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:45:12.779425 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:45:12.780002 (+ 577us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:45:12.780046 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:45:12.780064 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:45:12.780153 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0926 03:45:12.780297 (+ 144us) server_negotiation.cc:299] Negotiation successful
0926 03:45:12.780450 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":435,"thread_start_us":291,"threads_started":1}
I20260926 03:45:12.782011 10551 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:45:12.863305 779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu with pid 10429
I20260926 03:45:12.879231 779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu pid 10078
I20260926 03:45:12.880425 10139 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:45:12.981427 779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu with pid 10078
2026-09-26T03:45:12Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2454 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-26T03:45:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:45:13Z Disabled control of system clock
I20260926 03:45:13.054992 779 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
/tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790394221564982-779-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790394221564982-779-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790394221564982-779-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790394221564982-779-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.194.254:36749
--webserver_interface=127.0.194.254
--webserver_port=0
--builtin_ntp_servers=127.0.194.212:33463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.194.254:36749
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260926 03:45:13.397620 10575 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:45:13.397929 10575 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:45:13.398095 10575 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:45:13.405481 10575 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:45:13.405604 10575 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:45:13.405704 10575 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:45:13.405808 10575 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:45:13.405885 10575 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:45:13.415432 10575 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.194.212:33463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790394221564982-779-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790394221564982-779-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.194.254:36749
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.194.254:36749
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790394221564982-779-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.194.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790394221564982-779-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:45:13.416858 10575 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:45:13.418663 10575 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:45:13.430612 10580 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:45:13.430647 10583 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:45:13.430701 10581 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:45:13.431430 10575 server_base.cc:1034] running on GCE node
I20260926 03:45:13.432277 10575 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:45:13.433743 10575 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:45:13.434978 10575 hybrid_clock.cc:648] HybridClock initialized: now 1790394313434927 us; error 41 us; skew 500 ppm
I20260926 03:45:13.435340 10575 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 03:45:13.437703 10575 webserver.cc:460] Webserver started at http://127.0.194.254:45489/ using document root <none> and password file <none>
I20260926 03:45:13.438422 10575 fs_manager.cc:359] Metadata directory not provided
I20260926 03:45:13.438555 10575 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:45:13.438935 10575 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:45:13.441887 10575 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790394221564982-779-0/minicluster-data/master-0/data/instance:
uuid: "83be14f18277480b9d31f26fdd54141c"
format_stamp: "Formatted at 2026-09-26 03:45:13 on dist-test-slave-n2th"
server_key: "1dcbab8db73740cd5686d89b19cb8bea"
server_key_iv: "a60b2b3cf9259bdcbae4b3cbbfd31213"
server_key_version: "encryptionkey@0"
I20260926 03:45:13.442721 10575 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790394221564982-779-0/minicluster-data/master-0/wal/instance:
uuid: "83be14f18277480b9d31f26fdd54141c"
format_stamp: "Formatted at 2026-09-26 03:45:13 on dist-test-slave-n2th"
server_key: "1dcbab8db73740cd5686d89b19cb8bea"
server_key_iv: "a60b2b3cf9259bdcbae4b3cbbfd31213"
server_key_version: "encryptionkey@0"
I20260926 03:45:13.448167 10575 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260926 03:45:13.451786 10589 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:45:13.453158 10575 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 03:45:13.453352 10575 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790394221564982-779-0/minicluster-data/master-0/data,/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790394221564982-779-0/minicluster-data/master-0/wal
uuid: "83be14f18277480b9d31f26fdd54141c"
format_stamp: "Formatted at 2026-09-26 03:45:13 on dist-test-slave-n2th"
server_key: "1dcbab8db73740cd5686d89b19cb8bea"
server_key_iv: "a60b2b3cf9259bdcbae4b3cbbfd31213"
server_key_version: "encryptionkey@0"
I20260926 03:45:13.453593 10575 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790394221564982-779-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790394221564982-779-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790394221564982-779-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:45:13.459590 10575 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:45:13.460381 10575 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:45:13.460676 10575 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:45:13.478873 10575 rpc_server.cc:307] RPC server started. Bound to: 127.0.194.254:36749
I20260926 03:45:13.478936 10640 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.194.254:36749 every 8 connection(s)
I20260926 03:45:13.480270 10575 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790394221564982-779-0/minicluster-data/master-0/data/info.pb
I20260926 03:45:13.484066 10641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:45:13.487799 779 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu as pid 10575
I20260926 03:45:13.488022 779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790394221564982-779-0/minicluster-data/master-0/wal/instance
I20260926 03:45:13.488840 779 external_mini_cluster.cc:1428] Setting key 37e181a79d1d6ae77cacf2b133e1a1c0
I20260926 03:45:13.498149 10644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:45:13.490987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37100 (local address 127.0.194.254:36749)
0926 03:45:13.492099 (+ 1112us) server_negotiation.cc:206] Beginning negotiation
0926 03:45:13.492114 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 03:45:13.492169 (+ 55us) server_negotiation.cc:407] Connection header received
0926 03:45:13.493872 (+ 1703us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:45:13.493906 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:45:13.494508 (+ 602us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:45:13.495011 (+ 503us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:45:13.496067 (+ 1056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:45:13.496075 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:45:13.496097 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:45:13.496114 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:45:13.496187 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:45:13.496207 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:45:13.496442 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0926 03:45:13.496583 (+ 141us) server_negotiation.cc:299] Negotiation successful
0926 03:45:13.496865 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":586,"thread_start_us":386,"threads_started":1}
I20260926 03:45:13.498807 10641 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83be14f18277480b9d31f26fdd54141c: Bootstrap starting.
I20260926 03:45:13.502030 10641 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 83be14f18277480b9d31f26fdd54141c: Neither blocks nor log segments found. Creating new log.
I20260926 03:45:13.503050 10641 log.cc:824] T 00000000000000000000000000000000 P 83be14f18277480b9d31f26fdd54141c: Log is configured to *not* fsync() on all Append() calls
I20260926 03:45:13.505949 10641 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83be14f18277480b9d31f26fdd54141c: No bootstrap required, opened a new log
I20260926 03:45:13.513298 10641 raft_consensus.cc:348] T 00000000000000000000000000000000 P 83be14f18277480b9d31f26fdd54141c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83be14f18277480b9d31f26fdd54141c" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 36749 } }
I20260926 03:45:13.513752 10641 raft_consensus.cc:374] T 00000000000000000000000000000000 P 83be14f18277480b9d31f26fdd54141c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:45:13.513856 10641 raft_consensus.cc:729] T 00000000000000000000000000000000 P 83be14f18277480b9d31f26fdd54141c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83be14f18277480b9d31f26fdd54141c, State: Initialized, Role: FOLLOWER
I20260926 03:45:13.514405 10641 consensus_queue.cc:260] T 00000000000000000000000000000000 P 83be14f18277480b9d31f26fdd54141c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83be14f18277480b9d31f26fdd54141c" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 36749 } }
I20260926 03:45:13.514602 10641 raft_consensus.cc:388] T 00000000000000000000000000000000 P 83be14f18277480b9d31f26fdd54141c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:45:13.514695 10641 raft_consensus.cc:482] T 00000000000000000000000000000000 P 83be14f18277480b9d31f26fdd54141c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:45:13.514887 10641 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 83be14f18277480b9d31f26fdd54141c [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:45:13.517113 10641 raft_consensus.cc:504] T 00000000000000000000000000000000 P 83be14f18277480b9d31f26fdd54141c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83be14f18277480b9d31f26fdd54141c" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 36749 } }
I20260926 03:45:13.517661 10641 leader_election.cc:302] T 00000000000000000000000000000000 P 83be14f18277480b9d31f26fdd54141c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 83be14f18277480b9d31f26fdd54141c; no voters:
I20260926 03:45:13.518296 10641 leader_election.cc:288] T 00000000000000000000000000000000 P 83be14f18277480b9d31f26fdd54141c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:45:13.518443 10646 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 83be14f18277480b9d31f26fdd54141c [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:45:13.519214 10646 raft_consensus.cc:686] T 00000000000000000000000000000000 P 83be14f18277480b9d31f26fdd54141c [term 1 LEADER]: Becoming Leader. State: Replica: 83be14f18277480b9d31f26fdd54141c, State: Running, Role: LEADER
I20260926 03:45:13.519744 10646 consensus_queue.cc:237] T 00000000000000000000000000000000 P 83be14f18277480b9d31f26fdd54141c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83be14f18277480b9d31f26fdd54141c" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 36749 } }
I20260926 03:45:13.520401 10641 sys_catalog.cc:564] T 00000000000000000000000000000000 P 83be14f18277480b9d31f26fdd54141c [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:45:13.522591 10648 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83be14f18277480b9d31f26fdd54141c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 83be14f18277480b9d31f26fdd54141c. Latest consensus state: current_term: 1 leader_uuid: "83be14f18277480b9d31f26fdd54141c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83be14f18277480b9d31f26fdd54141c" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 36749 } } }
I20260926 03:45:13.523067 10648 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83be14f18277480b9d31f26fdd54141c [sys.catalog]: This master's current role is: LEADER
I20260926 03:45:13.523864 10647 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83be14f18277480b9d31f26fdd54141c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "83be14f18277480b9d31f26fdd54141c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83be14f18277480b9d31f26fdd54141c" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 36749 } } }
I20260926 03:45:13.524204 10647 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83be14f18277480b9d31f26fdd54141c [sys.catalog]: This master's current role is: LEADER
I20260926 03:45:13.526491 10655 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:45:13.530859 10655 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:45:13.539242 10655 catalog_manager.cc:1347] Generated new cluster ID: 68847b85e39b422e990294599653644b
I20260926 03:45:13.539373 10655 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:45:13.557626 10655 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:45:13.558139 10655 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:45:13.568557 10655 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 83be14f18277480b9d31f26fdd54141c: Generated new TSK 0
I20260926 03:45:13.569468 10655 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 03:45:13.581169 779 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
/tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790394221564982-779-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790394221564982-779-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790394221564982-779-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790394221564982-779-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.194.193:0
--local_ip_for_outbound_sockets=127.0.194.193
--webserver_interface=127.0.194.193
--webserver_port=0
--tserver_master_addrs=127.0.194.254:36749
--builtin_ntp_servers=127.0.194.212:33463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260926 03:45:13.912351 10665 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:45:13.912645 10665 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:45:13.912837 10665 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:45:13.920229 10665 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:45:13.920404 10665 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:45:13.920542 10665 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.194.193
I20260926 03:45:13.930128 10665 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.194.212:33463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790394221564982-779-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790394221564982-779-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.194.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790394221564982-779-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.194.193
--webserver_port=0
--tserver_master_addrs=127.0.194.254:36749
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.194.193
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790394221564982-779-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:45:13.931499 10665 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:45:13.933236 10665 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:45:13.946314 10670 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:45:13.946230 10671 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:45:13.946276 10673 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:45:13.947178 10665 server_base.cc:1034] running on GCE node
I20260926 03:45:13.947748 10665 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:45:13.948705 10665 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:45:13.949913 10665 hybrid_clock.cc:648] HybridClock initialized: now 1790394313949859 us; error 45 us; skew 500 ppm
I20260926 03:45:13.950253 10665 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 03:45:13.952776 10665 webserver.cc:460] Webserver started at http://127.0.194.193:42075/ using document root <none> and password file <none>
I20260926 03:45:13.953495 10665 fs_manager.cc:359] Metadata directory not provided
I20260926 03:45:13.953611 10665 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:45:13.953940 10665 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:45:13.958411 10665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790394221564982-779-0/minicluster-data/ts-0/data/instance:
uuid: "c88279d5b0a841239893877a5f68c69e"
format_stamp: "Formatted at 2026-09-26 03:45:13 on dist-test-slave-n2th"
server_key: "e8c283443f465945692121fe8eae875e"
server_key_iv: "d0d72e12d5bdf9960d35e9f82c78f579"
server_key_version: "encryptionkey@0"
I20260926 03:45:13.959337 10665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790394221564982-779-0/minicluster-data/ts-0/wal/instance:
uuid: "c88279d5b0a841239893877a5f68c69e"
format_stamp: "Formatted at 2026-09-26 03:45:13 on dist-test-slave-n2th"
server_key: "e8c283443f465945692121fe8eae875e"
server_key_iv: "d0d72e12d5bdf9960d35e9f82c78f579"
server_key_version: "encryptionkey@0"
I20260926 03:45:13.964551 10665 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 03:45:13.968025 10679 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:45:13.969405 10665 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 03:45:13.969586 10665 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790394221564982-779-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790394221564982-779-0/minicluster-data/ts-0/wal
uuid: "c88279d5b0a841239893877a5f68c69e"
format_stamp: "Formatted at 2026-09-26 03:45:13 on dist-test-slave-n2th"
server_key: "e8c283443f465945692121fe8eae875e"
server_key_iv: "d0d72e12d5bdf9960d35e9f82c78f579"
server_key_version: "encryptionkey@0"
I20260926 03:45:13.969826 10665 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790394221564982-779-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790394221564982-779-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790394221564982-779-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:45:13.975582 10665 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:45:13.976404 10665 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:45:13.976680 10665 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:45:13.977596 10665 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:45:13.979012 10665 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:45:13.979136 10665 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:45:13.979274 10665 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:45:13.979362 10665 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:45:14.014053 10665 rpc_server.cc:307] RPC server started. Bound to: 127.0.194.193:42135
I20260926 03:45:14.014106 10791 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.194.193:42135 every 8 connection(s)
I20260926 03:45:14.015475 10665 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790394221564982-779-0/minicluster-data/ts-0/data/info.pb
I20260926 03:45:14.023780 10794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:45:14.018059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.193:56081 (local address 127.0.194.254:36749)
0926 03:45:14.018503 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0926 03:45:14.018508 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:45:14.019776 (+ 1268us) server_negotiation.cc:407] Connection header received
0926 03:45:14.021544 (+ 1768us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:45:14.021550 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:45:14.021620 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:45:14.021735 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:45:14.023002 (+ 1267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:45:14.023008 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:45:14.023013 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:45:14.023031 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:45:14.023081 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:45:14.023092 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:45:14.023212 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0926 03:45:14.023413 (+ 201us) server_negotiation.cc:299] Negotiation successful
0926 03:45:14.023478 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":252,"threads_started":1}
I20260926 03:45:14.024125 779 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu as pid 10665
I20260926 03:45:14.024335 779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790394221564982-779-0/minicluster-data/ts-0/wal/instance
I20260926 03:45:14.024987 10795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:45:14.018703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:36749 (local address 127.0.194.193:56081)
0926 03:45:14.019629 (+ 926us) negotiation.cc:107] Waiting for socket to connect
0926 03:45:14.019668 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 03:45:14.021267 (+ 1599us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:45:14.021963 (+ 696us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:45:14.021972 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:45:14.022059 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:45:14.022647 (+ 588us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 03:45:14.022649 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:45:14.022685 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 03:45:14.022827 (+ 142us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:45:14.023184 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:45:14.023194 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:45:14.023201 (+ 7us) client_negotiation.cc:769] Sending connection context
0926 03:45:14.023477 (+ 276us) client_negotiation.cc:240] Negotiation successful
0926 03:45:14.023735 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":249,"threads_started":1}
I20260926 03:45:14.025072 779 external_mini_cluster.cc:1428] Setting key c2e8a96e156c736f430b0bd4a484ad74
I20260926 03:45:14.026612 10792 heartbeater.cc:347] Connected to a master server at 127.0.194.254:36749
I20260926 03:45:14.026961 10792 heartbeater.cc:464] Registering TS with master...
I20260926 03:45:14.027648 10792 heartbeater.cc:511] Master 127.0.194.254:36749 requested a full tablet report, sending...
I20260926 03:45:14.028012 779 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
/tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790394221564982-779-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790394221564982-779-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790394221564982-779-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790394221564982-779-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.194.194:0
--local_ip_for_outbound_sockets=127.0.194.194
--webserver_interface=127.0.194.194
--webserver_port=0
--tserver_master_addrs=127.0.194.254:36749
--builtin_ntp_servers=127.0.194.212:33463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260926 03:45:14.029665 10605 ts_manager.cc:194] Registered new tserver with Master: c88279d5b0a841239893877a5f68c69e (127.0.194.193:42135)
W20260926 03:45:14.357324 10796 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:45:14.357654 10796 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:45:14.357813 10796 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:45:14.365273 10796 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:45:14.365437 10796 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:45:14.365600 10796 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.194.194
I20260926 03:45:14.375111 10796 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.194.212:33463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790394221564982-779-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790394221564982-779-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.194.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790394221564982-779-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.194.194
--webserver_port=0
--tserver_master_addrs=127.0.194.254:36749
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.194.194
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790394221564982-779-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:45:14.376433 10796 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:45:14.378216 10796 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:45:14.391177 10801 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:45:14.391177 10802 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:45:14.391163 10804 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:45:14.392444 10796 server_base.cc:1034] running on GCE node
I20260926 03:45:14.393083 10796 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:45:14.394034 10796 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:45:14.395258 10796 hybrid_clock.cc:648] HybridClock initialized: now 1790394314395199 us; error 53 us; skew 500 ppm
I20260926 03:45:14.395578 10796 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 03:45:14.397848 10796 webserver.cc:460] Webserver started at http://127.0.194.194:35119/ using document root <none> and password file <none>
I20260926 03:45:14.398589 10796 fs_manager.cc:359] Metadata directory not provided
I20260926 03:45:14.398717 10796 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:45:14.399082 10796 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:45:14.401832 10796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790394221564982-779-0/minicluster-data/ts-1/data/instance:
uuid: "abc130e3bb7a430abae110754c39a48b"
format_stamp: "Formatted at 2026-09-26 03:45:14 on dist-test-slave-n2th"
server_key: "1da20d99ff93d01a74ae0dffa9c51ada"
server_key_iv: "972a383204079ef847f3058966b2e43a"
server_key_version: "encryptionkey@0"
I20260926 03:45:14.402547 10796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790394221564982-779-0/minicluster-data/ts-1/wal/instance:
uuid: "abc130e3bb7a430abae110754c39a48b"
format_stamp: "Formatted at 2026-09-26 03:45:14 on dist-test-slave-n2th"
server_key: "1da20d99ff93d01a74ae0dffa9c51ada"
server_key_iv: "972a383204079ef847f3058966b2e43a"
server_key_version: "encryptionkey@0"
I20260926 03:45:14.407586 10796 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260926 03:45:14.411042 10810 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:45:14.412395 10796 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 03:45:14.412571 10796 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790394221564982-779-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790394221564982-779-0/minicluster-data/ts-1/wal
uuid: "abc130e3bb7a430abae110754c39a48b"
format_stamp: "Formatted at 2026-09-26 03:45:14 on dist-test-slave-n2th"
server_key: "1da20d99ff93d01a74ae0dffa9c51ada"
server_key_iv: "972a383204079ef847f3058966b2e43a"
server_key_version: "encryptionkey@0"
I20260926 03:45:14.412788 10796 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790394221564982-779-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790394221564982-779-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790394221564982-779-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:45:14.418409 10796 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:45:14.419363 10796 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:45:14.419626 10796 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:45:14.420523 10796 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:45:14.422053 10796 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:45:14.422170 10796 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:45:14.422303 10796 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:45:14.422386 10796 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:45:14.455344 10796 rpc_server.cc:307] RPC server started. Bound to: 127.0.194.194:34145
I20260926 03:45:14.455407 10922 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.194.194:34145 every 8 connection(s)
I20260926 03:45:14.456676 10796 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790394221564982-779-0/minicluster-data/ts-1/data/info.pb
I20260926 03:45:14.460121 779 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu as pid 10796
I20260926 03:45:14.460325 779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790394221564982-779-0/minicluster-data/ts-1/wal/instance
I20260926 03:45:14.460762 779 external_mini_cluster.cc:1428] Setting key 378827b3d5b9fa305e8427d583ef30f0
I20260926 03:45:14.463809 779 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
/tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790394221564982-779-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790394221564982-779-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790394221564982-779-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790394221564982-779-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.194.195:0
--local_ip_for_outbound_sockets=127.0.194.195
--webserver_interface=127.0.194.195
--webserver_port=0
--tserver_master_addrs=127.0.194.254:36749
--builtin_ntp_servers=127.0.194.212:33463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260926 03:45:14.465327 10794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:45:14.459408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.194:40117 (local address 127.0.194.254:36749)
0926 03:45:14.459579 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0926 03:45:14.459583 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:45:14.461093 (+ 1510us) server_negotiation.cc:407] Connection header received
0926 03:45:14.463002 (+ 1909us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:45:14.463045 (+ 43us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:45:14.463119 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:45:14.463274 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:45:14.464660 (+ 1386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:45:14.464664 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:45:14.464668 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:45:14.464682 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:45:14.464713 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:45:14.464723 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:45:14.464813 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 03:45:14.465054 (+ 241us) server_negotiation.cc:299] Negotiation successful
0926 03:45:14.465102 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 03:45:14.466820 10925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:45:14.459847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:36749 (local address 127.0.194.194:40117)
0926 03:45:14.460916 (+ 1069us) negotiation.cc:107] Waiting for socket to connect
0926 03:45:14.460967 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 03:45:14.462675 (+ 1708us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:45:14.463459 (+ 784us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:45:14.463468 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:45:14.463562 (+ 94us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:45:14.464307 (+ 745us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 03:45:14.464313 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:45:14.464367 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 03:45:14.464486 (+ 119us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:45:14.464822 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:45:14.464843 (+ 21us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:45:14.464855 (+ 12us) client_negotiation.cc:769] Sending connection context
0926 03:45:14.465149 (+ 294us) client_negotiation.cc:240] Negotiation successful
0926 03:45:14.465550 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":339,"threads_started":1}
I20260926 03:45:14.468359 10923 heartbeater.cc:347] Connected to a master server at 127.0.194.254:36749
I20260926 03:45:14.468673 10923 heartbeater.cc:464] Registering TS with master...
I20260926 03:45:14.469383 10923 heartbeater.cc:511] Master 127.0.194.254:36749 requested a full tablet report, sending...
I20260926 03:45:14.470897 10605 ts_manager.cc:194] Registered new tserver with Master: abc130e3bb7a430abae110754c39a48b (127.0.194.194:34145)
W20260926 03:45:14.796917 10926 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:45:14.797215 10926 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:45:14.797318 10926 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:45:14.804658 10926 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:45:14.804773 10926 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:45:14.804886 10926 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.194.195
I20260926 03:45:14.814059 10926 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.194.212:33463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790394221564982-779-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790394221564982-779-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.194.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790394221564982-779-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.194.195
--webserver_port=0
--tserver_master_addrs=127.0.194.254:36749
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.194.195
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790394221564982-779-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:45:14.815371 10926 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:45:14.817411 10926 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:45:14.831804 10934 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:45:14.831881 10932 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:45:14.831847 10931 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:45:14.833392 10926 server_base.cc:1034] running on GCE node
I20260926 03:45:14.834072 10926 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:45:14.835108 10926 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:45:14.836323 10926 hybrid_clock.cc:648] HybridClock initialized: now 1790394314836257 us; error 56 us; skew 500 ppm
I20260926 03:45:14.836678 10926 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 03:45:14.839154 10926 webserver.cc:460] Webserver started at http://127.0.194.195:45195/ using document root <none> and password file <none>
I20260926 03:45:14.839888 10926 fs_manager.cc:359] Metadata directory not provided
I20260926 03:45:14.839980 10926 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:45:14.840255 10926 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:45:14.843218 10926 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790394221564982-779-0/minicluster-data/ts-2/data/instance:
uuid: "3b2387c8aa0544788bdfac0ca2b7fb9b"
format_stamp: "Formatted at 2026-09-26 03:45:14 on dist-test-slave-n2th"
server_key: "f764d275e6a6e7eea7811639df7d6551"
server_key_iv: "6c3ae4503bb05c5cb2f030f308c000aa"
server_key_version: "encryptionkey@0"
I20260926 03:45:14.843984 10926 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790394221564982-779-0/minicluster-data/ts-2/wal/instance:
uuid: "3b2387c8aa0544788bdfac0ca2b7fb9b"
format_stamp: "Formatted at 2026-09-26 03:45:14 on dist-test-slave-n2th"
server_key: "f764d275e6a6e7eea7811639df7d6551"
server_key_iv: "6c3ae4503bb05c5cb2f030f308c000aa"
server_key_version: "encryptionkey@0"
I20260926 03:45:14.849366 10926 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260926 03:45:14.852943 10940 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:45:14.854347 10926 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 03:45:14.854552 10926 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790394221564982-779-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790394221564982-779-0/minicluster-data/ts-2/wal
uuid: "3b2387c8aa0544788bdfac0ca2b7fb9b"
format_stamp: "Formatted at 2026-09-26 03:45:14 on dist-test-slave-n2th"
server_key: "f764d275e6a6e7eea7811639df7d6551"
server_key_iv: "6c3ae4503bb05c5cb2f030f308c000aa"
server_key_version: "encryptionkey@0"
I20260926 03:45:14.854841 10926 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790394221564982-779-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790394221564982-779-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790394221564982-779-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:45:14.860808 10926 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:45:14.861747 10926 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:45:14.862042 10926 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:45:14.863049 10926 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:45:14.864486 10926 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:45:14.864612 10926 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:45:14.864737 10926 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:45:14.864821 10926 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:45:14.898277 10926 rpc_server.cc:307] RPC server started. Bound to: 127.0.194.195:41949
I20260926 03:45:14.898327 11052 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.194.195:41949 every 8 connection(s)
I20260926 03:45:14.899678 10926 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790394221564982-779-0/minicluster-data/ts-2/data/info.pb
I20260926 03:45:14.907562 10794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:45:14.902193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.195:59509 (local address 127.0.194.254:36749)
0926 03:45:14.902411 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0926 03:45:14.902417 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:45:14.903863 (+ 1446us) server_negotiation.cc:407] Connection header received
0926 03:45:14.905482 (+ 1619us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:45:14.905486 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:45:14.905539 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:45:14.905628 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:45:14.906935 (+ 1307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:45:14.906940 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:45:14.906949 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:45:14.906965 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:45:14.906997 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:45:14.907008 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:45:14.907126 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0926 03:45:14.907289 (+ 163us) server_negotiation.cc:299] Negotiation successful
0926 03:45:14.907340 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260926 03:45:14.908413 779 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu as pid 10926
I20260926 03:45:14.908601 779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790394221564982-779-0/minicluster-data/ts-2/wal/instance
I20260926 03:45:14.908752 11055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:45:14.902805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:36749 (local address 127.0.194.195:59509)
0926 03:45:14.903704 (+ 899us) negotiation.cc:107] Waiting for socket to connect
0926 03:45:14.903738 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0926 03:45:14.905229 (+ 1491us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:45:14.905836 (+ 607us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:45:14.905844 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:45:14.905949 (+ 105us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:45:14.906581 (+ 632us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 03:45:14.906583 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:45:14.906621 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 03:45:14.906704 (+ 83us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:45:14.907105 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:45:14.907113 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:45:14.907118 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 03:45:14.907337 (+ 219us) client_negotiation.cc:240] Negotiation successful
0926 03:45:14.907543 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":265,"threads_started":1}
I20260926 03:45:14.909135 779 external_mini_cluster.cc:1428] Setting key dd4ef85fcc8ccdc48dab3c13f5574f7b
I20260926 03:45:14.910355 11053 heartbeater.cc:347] Connected to a master server at 127.0.194.254:36749
I20260926 03:45:14.910694 11053 heartbeater.cc:464] Registering TS with master...
I20260926 03:45:14.911335 11053 heartbeater.cc:511] Master 127.0.194.254:36749 requested a full tablet report, sending...
I20260926 03:45:14.912679 10605 ts_manager.cc:194] Registered new tserver with Master: 3b2387c8aa0544788bdfac0ca2b7fb9b (127.0.194.195:41949)
I20260926 03:45:14.924619 779 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260926 03:45:14.930783 10794 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:45:14.929244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37116 (local address 127.0.194.254:36749)
0926 03:45:14.929405 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0926 03:45:14.929409 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:45:14.929886 (+ 477us) server_negotiation.cc:407] Connection header received
0926 03:45:14.930024 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:45:14.930028 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:45:14.930078 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:45:14.930167 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:45:14.930473 (+ 306us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37116: BlockingRecv error: recv got EOF from 127.0.0.1:37116 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 03:45:14.931712 779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu pid 10665
I20260926 03:45:14.935154 11062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:45:14.932658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36400 (local address 127.0.194.193:42135)
0926 03:45:14.933185 (+ 527us) server_negotiation.cc:206] Beginning negotiation
0926 03:45:14.933201 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 03:45:14.933226 (+ 25us) server_negotiation.cc:407] Connection header received
0926 03:45:14.933282 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:45:14.933289 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:45:14.933512 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:45:14.933638 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:45:14.934498 (+ 860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:45:14.934505 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:45:14.934517 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:45:14.934535 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:45:14.934577 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:45:14.934591 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:45:14.934692 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0926 03:45:14.934836 (+ 144us) server_negotiation.cc:299] Negotiation successful
0926 03:45:14.934928 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":388,"thread_start_us":276,"threads_started":1}
I20260926 03:45:14.936343 10787 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:45:15.017956 779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu with pid 10665
I20260926 03:45:15.034862 779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu pid 10796
I20260926 03:45:15.038632 11064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:45:15.035891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39364 (local address 127.0.194.194:34145)
0926 03:45:15.036440 (+ 549us) server_negotiation.cc:206] Beginning negotiation
0926 03:45:15.036447 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:45:15.036475 (+ 28us) server_negotiation.cc:407] Connection header received
0926 03:45:15.036542 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:45:15.036547 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:45:15.036775 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:45:15.036909 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:45:15.037856 (+ 947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:45:15.037864 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:45:15.037878 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:45:15.037895 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:45:15.037948 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:45:15.037965 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:45:15.038092 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0926 03:45:15.038154 (+ 62us) server_negotiation.cc:299] Negotiation successful
0926 03:45:15.038336 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":420,"thread_start_us":299,"threads_started":1}
I20260926 03:45:15.040019 10918 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:45:15.122201 779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu with pid 10796
I20260926 03:45:15.140040 779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu pid 10926
I20260926 03:45:15.144204 11066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:45:15.141119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33304 (local address 127.0.194.195:41949)
0926 03:45:15.141754 (+ 635us) server_negotiation.cc:206] Beginning negotiation
0926 03:45:15.141761 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:45:15.141791 (+ 30us) server_negotiation.cc:407] Connection header received
0926 03:45:15.141852 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:45:15.141859 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:45:15.142069 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:45:15.142245 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:45:15.143440 (+ 1195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:45:15.143446 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:45:15.143458 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:45:15.143476 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:45:15.143514 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:45:15.143526 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:45:15.143712 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0926 03:45:15.143813 (+ 101us) server_negotiation.cc:299] Negotiation successful
0926 03:45:15.143911 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":481,"thread_start_us":324,"threads_started":1}
I20260926 03:45:15.145502 11048 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:45:15.226838 779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu with pid 10926
I20260926 03:45:15.243335 779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu pid 10575
I20260926 03:45:15.244675 10636 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:45:15.362012 779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu with pid 10575
2026-09-26T03:45:15Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2378 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790394221564982-779-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:45:15 dist-test-slave-n2th krb5kdc[11071](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:45:15 dist-test-slave-n2th krb5kdc[11071](info): set up 2 sockets
Sep 26 03:45:15 dist-test-slave-n2th krb5kdc[11071](info): commencing operation
krb5kdc: starting...
W20260926 03:45:16.467777 779 mini_kdc.cc:121] Time spent starting KDC: real 1.065s user 0.005s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790394221564982-779-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790394221564982-779-0/krb5kdc/test-user.keytab.
Sep 26 03:45:16 dist-test-slave-n2th krb5kdc[11071](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394316, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:45:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:45:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.194.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.194.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.194.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790394221564982-779-0/krb5kdc/kudu_127.0.194.254.keytab.
Entry for principal kudu/127.0.194.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790394221564982-779-0/krb5kdc/kudu_127.0.194.254.keytab.
I20260926 03:45:16.706056 779 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
/tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790394221564982-779-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790394221564982-779-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790394221564982-779-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790394221564982-779-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.194.254:41957
--webserver_interface=127.0.194.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790394221564982-779-0/krb5kdc/kudu_127.0.194.254.keytab
--principal=kudu/127.0.194.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.194.212:37267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.194.254:41957
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790394221564982-779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790394221564982-779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790394221564982-779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:45:17.039074 11084 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:45:17.039430 11084 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:45:17.039595 11084 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:45:17.046957 11084 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:45:17.047077 11084 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:45:17.047122 11084 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:45:17.047209 11084 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:45:17.047277 11084 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:45:17.057235 11084 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.194.212:37267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790394221564982-779-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790394221564982-779-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.194.254:41957
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790394221564982-779-0/krb5kdc/kudu_127.0.194.254.keytab
--principal=kudu/127.0.194.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.194.254:41957
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790394221564982-779-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.194.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790394221564982-779-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:45:17.058632 11084 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:45:17.060002 11084 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:45:17.072081 11090 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:45:17.072191 11092 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:45:17.072047 11089 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:45:17.073000 11084 server_base.cc:1034] running on GCE node
I20260926 03:45:17.073829 11084 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:45:17.075409 11084 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:45:17.076663 11084 hybrid_clock.cc:648] HybridClock initialized: now 1790394317076614 us; error 39 us; skew 500 ppm
I20260926 03:45:17.077051 11084 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:45:17 dist-test-slave-n2th krb5kdc[11071](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394317, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:45:17.084550 11084 init.cc:377] Logged in from keytab as kudu/127.0.194.254@KRBTEST.COM (short username kudu)
I20260926 03:45:17.086220 11084 webserver.cc:460] Webserver started at http://127.0.194.254:39763/ using document root <none> and password file <none>
I20260926 03:45:17.087101 11084 fs_manager.cc:359] Metadata directory not provided
I20260926 03:45:17.087256 11084 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:45:17.087567 11084 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:45:17.090592 11084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790394221564982-779-0/minicluster-data/master-0/data/instance:
uuid: "e34a4090276f4d659f9290a3494c90b3"
format_stamp: "Formatted at 2026-09-26 03:45:17 on dist-test-slave-n2th"
server_key: "21944e6cb7d27c798aff73cc7affa18b"
server_key_iv: "6f1615f07ffee5999c0e1a8d7bab4ce7"
server_key_version: "encryptionkey@0"
I20260926 03:45:17.091441 11084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790394221564982-779-0/minicluster-data/master-0/wal/instance:
uuid: "e34a4090276f4d659f9290a3494c90b3"
format_stamp: "Formatted at 2026-09-26 03:45:17 on dist-test-slave-n2th"
server_key: "21944e6cb7d27c798aff73cc7affa18b"
server_key_iv: "6f1615f07ffee5999c0e1a8d7bab4ce7"
server_key_version: "encryptionkey@0"
I20260926 03:45:17.096848 11084 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 03:45:17.100517 11099 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:45:17.102108 11084 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 03:45:17.102313 11084 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790394221564982-779-0/minicluster-data/master-0/data,/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790394221564982-779-0/minicluster-data/master-0/wal
uuid: "e34a4090276f4d659f9290a3494c90b3"
format_stamp: "Formatted at 2026-09-26 03:45:17 on dist-test-slave-n2th"
server_key: "21944e6cb7d27c798aff73cc7affa18b"
server_key_iv: "6f1615f07ffee5999c0e1a8d7bab4ce7"
server_key_version: "encryptionkey@0"
I20260926 03:45:17.102530 11084 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790394221564982-779-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790394221564982-779-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790394221564982-779-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:45:17.121425 11084 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:45:17.126785 11084 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:45:17.127107 11084 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:45:17.145473 11084 rpc_server.cc:307] RPC server started. Bound to: 127.0.194.254:41957
I20260926 03:45:17.145475 11150 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.194.254:41957 every 8 connection(s)
I20260926 03:45:17.146975 11084 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790394221564982-779-0/minicluster-data/master-0/data/info.pb
I20260926 03:45:17.149350 779 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu as pid 11084
I20260926 03:45:17.149593 779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790394221564982-779-0/minicluster-data/master-0/wal/instance
I20260926 03:45:17.150240 779 external_mini_cluster.cc:1428] Setting key 0bbe64469df85653a0d559e650d58ba1
I20260926 03:45:17.151186 11151 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 26 03:45:17 dist-test-slave-n2th krb5kdc[11071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394316, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.194.254@KRBTEST.COM
I20260926 03:45:17.165442 11151 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e34a4090276f4d659f9290a3494c90b3: Bootstrap starting.
I20260926 03:45:17.169808 11151 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e34a4090276f4d659f9290a3494c90b3: Neither blocks nor log segments found. Creating new log.
I20260926 03:45:17.171197 11151 log.cc:824] T 00000000000000000000000000000000 P e34a4090276f4d659f9290a3494c90b3: Log is configured to *not* fsync() on all Append() calls
I20260926 03:45:17.174688 11151 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e34a4090276f4d659f9290a3494c90b3: No bootstrap required, opened a new log
I20260926 03:45:17.174957 11154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:45:17.152316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40870 (local address 127.0.194.254:41957)
0926 03:45:17.153125 (+ 809us) server_negotiation.cc:206] Beginning negotiation
0926 03:45:17.153140 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 03:45:17.153190 (+ 50us) server_negotiation.cc:407] Connection header received
0926 03:45:17.154940 (+ 1750us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:45:17.154968 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:45:17.155520 (+ 552us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:45:17.156003 (+ 483us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:45:17.157432 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:45:17.159572 (+ 2140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:45:17.160876 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:45:17.161959 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:45:17.162152 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:45:17.166227 (+ 4075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:45:17.166239 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:45:17.166260 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:45:17.166317 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:45:17.170362 (+ 4045us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:45:17.171169 (+ 807us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:45:17.171179 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:45:17.171195 (+ 16us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:45:17.171409 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:45:17.171811 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:45:17.171819 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:45:17.171830 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:45:17.172444 (+ 614us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:45:17.172774 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0926 03:45:17.173125 (+ 351us) server_negotiation.cc:299] Negotiation successful
0926 03:45:17.173550 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":559,"thread_start_us":311,"threads_started":1}
I20260926 03:45:17.183038 11151 raft_consensus.cc:348] T 00000000000000000000000000000000 P e34a4090276f4d659f9290a3494c90b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e34a4090276f4d659f9290a3494c90b3" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 41957 } }
I20260926 03:45:17.183408 11151 raft_consensus.cc:374] T 00000000000000000000000000000000 P e34a4090276f4d659f9290a3494c90b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:45:17.183490 11151 raft_consensus.cc:729] T 00000000000000000000000000000000 P e34a4090276f4d659f9290a3494c90b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e34a4090276f4d659f9290a3494c90b3, State: Initialized, Role: FOLLOWER
I20260926 03:45:17.183995 11151 consensus_queue.cc:260] T 00000000000000000000000000000000 P e34a4090276f4d659f9290a3494c90b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e34a4090276f4d659f9290a3494c90b3" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 41957 } }
I20260926 03:45:17.184198 11151 raft_consensus.cc:388] T 00000000000000000000000000000000 P e34a4090276f4d659f9290a3494c90b3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:45:17.184296 11151 raft_consensus.cc:482] T 00000000000000000000000000000000 P e34a4090276f4d659f9290a3494c90b3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:45:17.184424 11151 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e34a4090276f4d659f9290a3494c90b3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:45:17.186553 11151 raft_consensus.cc:504] T 00000000000000000000000000000000 P e34a4090276f4d659f9290a3494c90b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e34a4090276f4d659f9290a3494c90b3" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 41957 } }
I20260926 03:45:17.187213 11151 leader_election.cc:302] T 00000000000000000000000000000000 P e34a4090276f4d659f9290a3494c90b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e34a4090276f4d659f9290a3494c90b3; no voters:
I20260926 03:45:17.187836 11151 leader_election.cc:288] T 00000000000000000000000000000000 P e34a4090276f4d659f9290a3494c90b3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:45:17.187968 11156 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e34a4090276f4d659f9290a3494c90b3 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:45:17.188678 11156 raft_consensus.cc:686] T 00000000000000000000000000000000 P e34a4090276f4d659f9290a3494c90b3 [term 1 LEADER]: Becoming Leader. State: Replica: e34a4090276f4d659f9290a3494c90b3, State: Running, Role: LEADER
I20260926 03:45:17.189203 11156 consensus_queue.cc:237] T 00000000000000000000000000000000 P e34a4090276f4d659f9290a3494c90b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e34a4090276f4d659f9290a3494c90b3" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 41957 } }
I20260926 03:45:17.189982 11151 sys_catalog.cc:564] T 00000000000000000000000000000000 P e34a4090276f4d659f9290a3494c90b3 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:45:17.192392 11158 sys_catalog.cc:455] T 00000000000000000000000000000000 P e34a4090276f4d659f9290a3494c90b3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e34a4090276f4d659f9290a3494c90b3. Latest consensus state: current_term: 1 leader_uuid: "e34a4090276f4d659f9290a3494c90b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e34a4090276f4d659f9290a3494c90b3" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 41957 } } }
I20260926 03:45:17.192730 11158 sys_catalog.cc:458] T 00000000000000000000000000000000 P e34a4090276f4d659f9290a3494c90b3 [sys.catalog]: This master's current role is: LEADER
I20260926 03:45:17.194238 11163 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:45:17.194139 11157 sys_catalog.cc:455] T 00000000000000000000000000000000 P e34a4090276f4d659f9290a3494c90b3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e34a4090276f4d659f9290a3494c90b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e34a4090276f4d659f9290a3494c90b3" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 41957 } } }
I20260926 03:45:17.194474 11157 sys_catalog.cc:458] T 00000000000000000000000000000000 P e34a4090276f4d659f9290a3494c90b3 [sys.catalog]: This master's current role is: LEADER
I20260926 03:45:17.199398 11163 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:45:17.209045 11163 catalog_manager.cc:1347] Generated new cluster ID: a0ce772d3e8c4bdd8a74c81d916e556c
I20260926 03:45:17.209190 11163 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:45:17.216691 11163 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:45:17.217980 11163 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:45:17.227311 11163 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e34a4090276f4d659f9290a3494c90b3: Generated new TSK 0
I20260926 03:45:17.228332 11163 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.194.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.194.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.194.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790394221564982-779-0/krb5kdc/kudu_127.0.194.193.keytab.
Entry for principal kudu/127.0.194.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790394221564982-779-0/krb5kdc/kudu_127.0.194.193.keytab.
I20260926 03:45:17.303110 779 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
/tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790394221564982-779-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790394221564982-779-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790394221564982-779-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790394221564982-779-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.194.193:0
--local_ip_for_outbound_sockets=127.0.194.193
--webserver_interface=127.0.194.193
--webserver_port=0
--tserver_master_addrs=127.0.194.254:41957
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790394221564982-779-0/krb5kdc/kudu_127.0.194.193.keytab
--principal=kudu/127.0.194.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.194.212:37267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790394221564982-779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790394221564982-779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790394221564982-779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:45:17.653688 11177 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:45:17.654000 11177 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:45:17.654094 11177 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:45:17.661475 11177 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:45:17.661603 11177 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:45:17.661729 11177 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.194.193
I20260926 03:45:17.672194 11177 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.194.212:37267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790394221564982-779-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790394221564982-779-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790394221564982-779-0/krb5kdc/kudu_127.0.194.193.keytab
--principal=kudu/127.0.194.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.194.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790394221564982-779-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.194.193
--webserver_port=0
--tserver_master_addrs=127.0.194.254:41957
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.194.193
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790394221564982-779-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:45:17.673518 11177 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:45:17.674985 11177 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:45:17.689034 11185 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:45:17.689085 11182 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:45:17.689110 11183 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:45:17.695555 11177 server_base.cc:1034] running on GCE node
I20260926 03:45:17.696281 11177 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:45:17.697403 11177 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:45:17.698609 11177 hybrid_clock.cc:648] HybridClock initialized: now 1790394317698540 us; error 67 us; skew 500 ppm
I20260926 03:45:17.698959 11177 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:45:17 dist-test-slave-n2th krb5kdc[11071](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394317, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:45:17.702210 11177 init.cc:377] Logged in from keytab as kudu/127.0.194.193@KRBTEST.COM (short username kudu)
I20260926 03:45:17.703855 11177 webserver.cc:460] Webserver started at http://127.0.194.193:40705/ using document root <none> and password file <none>
I20260926 03:45:17.704624 11177 fs_manager.cc:359] Metadata directory not provided
I20260926 03:45:17.704770 11177 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:45:17.705072 11177 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:45:17.708333 11177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790394221564982-779-0/minicluster-data/ts-0/data/instance:
uuid: "c35c4f35e08e4b3dbf9a9c25275ed908"
format_stamp: "Formatted at 2026-09-26 03:45:17 on dist-test-slave-n2th"
server_key: "3c8a03e05f18f6ec186106ec0bfb067c"
server_key_iv: "75039d9ee167f1577bf1aa99fd1b7fd7"
server_key_version: "encryptionkey@0"
I20260926 03:45:17.709118 11177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790394221564982-779-0/minicluster-data/ts-0/wal/instance:
uuid: "c35c4f35e08e4b3dbf9a9c25275ed908"
format_stamp: "Formatted at 2026-09-26 03:45:17 on dist-test-slave-n2th"
server_key: "3c8a03e05f18f6ec186106ec0bfb067c"
server_key_iv: "75039d9ee167f1577bf1aa99fd1b7fd7"
server_key_version: "encryptionkey@0"
I20260926 03:45:17.716951 11177 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.008s sys 0.000s
I20260926 03:45:17.721053 11192 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:45:17.722460 11177 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 03:45:17.722653 11177 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790394221564982-779-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790394221564982-779-0/minicluster-data/ts-0/wal
uuid: "c35c4f35e08e4b3dbf9a9c25275ed908"
format_stamp: "Formatted at 2026-09-26 03:45:17 on dist-test-slave-n2th"
server_key: "3c8a03e05f18f6ec186106ec0bfb067c"
server_key_iv: "75039d9ee167f1577bf1aa99fd1b7fd7"
server_key_version: "encryptionkey@0"
I20260926 03:45:17.722963 11177 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790394221564982-779-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790394221564982-779-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790394221564982-779-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:45:17.744863 11177 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:45:17.749308 11177 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:45:17.749616 11177 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:45:17.750672 11177 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:45:17.752055 11177 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:45:17.752162 11177 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:45:17.752311 11177 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:45:17.752382 11177 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:45:17.785775 11177 rpc_server.cc:307] RPC server started. Bound to: 127.0.194.193:41163
I20260926 03:45:17.785813 11304 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.194.193:41163 every 8 connection(s)
I20260926 03:45:17.787129 11177 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790394221564982-779-0/minicluster-data/ts-0/data/info.pb
I20260926 03:45:17.791631 779 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu as pid 11177
I20260926 03:45:17.791879 779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790394221564982-779-0/minicluster-data/ts-0/wal/instance
I20260926 03:45:17.792670 779 external_mini_cluster.cc:1428] Setting key 16a029ca7532dcc6324b2cc621d12c56
Sep 26 03:45:17 dist-test-slave-n2th krb5kdc[11071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394317, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.193@KRBTEST.COM for kudu/127.0.194.254@KRBTEST.COM
I20260926 03:45:17.810410 11307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:45:17.789661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.193:33459 (local address 127.0.194.254:41957)
0926 03:45:17.790159 (+ 498us) server_negotiation.cc:206] Beginning negotiation
0926 03:45:17.790164 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:45:17.791313 (+ 1149us) server_negotiation.cc:407] Connection header received
0926 03:45:17.793767 (+ 2454us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:45:17.793772 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:45:17.793856 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:45:17.793989 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:45:17.796000 (+ 2011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:45:17.796866 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:45:17.799375 (+ 2509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:45:17.799761 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:45:17.799859 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:45:17.804273 (+ 4414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:45:17.804282 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:45:17.804290 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:45:17.804330 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:45:17.807030 (+ 2700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:45:17.808015 (+ 985us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:45:17.808019 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:45:17.808022 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:45:17.808150 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:45:17.808646 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:45:17.808649 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:45:17.808651 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:45:17.808912 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:45:17.809020 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0926 03:45:17.809982 (+ 962us) server_negotiation.cc:299] Negotiation successful
0926 03:45:17.810160 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":391,"thread_start_us":278,"threads_started":1}
I20260926 03:45:17.812050 11308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:45:17.790194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:41957 (local address 127.0.194.193:33459)
0926 03:45:17.790969 (+ 775us) negotiation.cc:107] Waiting for socket to connect
0926 03:45:17.791029 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0926 03:45:17.793495 (+ 2466us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:45:17.794240 (+ 745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:45:17.794248 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:45:17.794977 (+ 729us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:45:17.795820 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:45:17.795849 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:45:17.797031 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:45:17.797039 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:45:17.799222 (+ 2183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:45:17.799240 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:45:17.800001 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:45:17.800010 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:45:17.800363 (+ 353us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:45:17.801117 (+ 754us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:45:17.801158 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:45:17.803990 (+ 2832us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:45:17.807181 (+ 3191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:45:17.807193 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:45:17.807214 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:45:17.807887 (+ 673us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:45:17.808256 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:45:17.808263 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:45:17.808269 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:45:17.808527 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:45:17.809041 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:45:17.809052 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:45:17.809567 (+ 515us) client_negotiation.cc:769] Sending connection context
0926 03:45:17.809888 (+ 321us) client_negotiation.cc:240] Negotiation successful
0926 03:45:17.810488 (+ 600us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":260,"threads_started":1}
I20260926 03:45:17.813623 11305 heartbeater.cc:347] Connected to a master server at 127.0.194.254:41957
I20260926 03:45:17.813956 11305 heartbeater.cc:464] Registering TS with master...
I20260926 03:45:17.814728 11305 heartbeater.cc:511] Master 127.0.194.254:41957 requested a full tablet report, sending...
I20260926 03:45:17.816916 11116 ts_manager.cc:194] Registered new tserver with Master: c35c4f35e08e4b3dbf9a9c25275ed908 (127.0.194.193:41163)
WARNING: no policy specified for kudu/127.0.194.194@KRBTEST.COM; defaulting to no policy
I20260926 03:45:17.818341 11116 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.194.193@KRBTEST.COM'} at 127.0.194.193:33459
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.194.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.194.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790394221564982-779-0/krb5kdc/kudu_127.0.194.194.keytab.
Entry for principal kudu/127.0.194.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790394221564982-779-0/krb5kdc/kudu_127.0.194.194.keytab.
I20260926 03:45:17.847829 779 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
/tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790394221564982-779-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790394221564982-779-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790394221564982-779-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790394221564982-779-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.194.194:0
--local_ip_for_outbound_sockets=127.0.194.194
--webserver_interface=127.0.194.194
--webserver_port=0
--tserver_master_addrs=127.0.194.254:41957
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790394221564982-779-0/krb5kdc/kudu_127.0.194.194.keytab
--principal=kudu/127.0.194.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.194.212:37267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790394221564982-779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790394221564982-779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790394221564982-779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:45:18.182876 11311 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:45:18.183223 11311 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:45:18.183403 11311 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:45:18.190884 11311 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:45:18.191044 11311 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:45:18.191237 11311 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.194.194
I20260926 03:45:18.201395 11311 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.194.212:37267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790394221564982-779-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790394221564982-779-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790394221564982-779-0/krb5kdc/kudu_127.0.194.194.keytab
--principal=kudu/127.0.194.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.194.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790394221564982-779-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.194.194
--webserver_port=0
--tserver_master_addrs=127.0.194.254:41957
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.194.194
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790394221564982-779-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:45:18.202867 11311 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:45:18.204232 11311 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:45:18.218694 11316 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:45:18.218791 11319 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:45:18.218755 11317 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:45:18.219421 11311 server_base.cc:1034] running on GCE node
I20260926 03:45:18.220121 11311 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:45:18.221136 11311 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:45:18.222368 11311 hybrid_clock.cc:648] HybridClock initialized: now 1790394318222307 us; error 60 us; skew 500 ppm
I20260926 03:45:18.222693 11311 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:45:18 dist-test-slave-n2th krb5kdc[11071](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394318, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:45:18.225973 11311 init.cc:377] Logged in from keytab as kudu/127.0.194.194@KRBTEST.COM (short username kudu)
I20260926 03:45:18.227545 11311 webserver.cc:460] Webserver started at http://127.0.194.194:41501/ using document root <none> and password file <none>
I20260926 03:45:18.228329 11311 fs_manager.cc:359] Metadata directory not provided
I20260926 03:45:18.228475 11311 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:45:18.228785 11311 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:45:18.231927 11311 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790394221564982-779-0/minicluster-data/ts-1/data/instance:
uuid: "0adabfbeaccc4d208b18dd145d3805e7"
format_stamp: "Formatted at 2026-09-26 03:45:18 on dist-test-slave-n2th"
server_key: "2c616df07164c9ed0a19ddfdc0bab2de"
server_key_iv: "61843e3dab458d20dd5eb2e6d1c3ad03"
server_key_version: "encryptionkey@0"
I20260926 03:45:18.232712 11311 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790394221564982-779-0/minicluster-data/ts-1/wal/instance:
uuid: "0adabfbeaccc4d208b18dd145d3805e7"
format_stamp: "Formatted at 2026-09-26 03:45:18 on dist-test-slave-n2th"
server_key: "2c616df07164c9ed0a19ddfdc0bab2de"
server_key_iv: "61843e3dab458d20dd5eb2e6d1c3ad03"
server_key_version: "encryptionkey@0"
I20260926 03:45:18.237938 11311 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 03:45:18.241639 11326 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:45:18.243250 11311 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 03:45:18.243438 11311 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790394221564982-779-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790394221564982-779-0/minicluster-data/ts-1/wal
uuid: "0adabfbeaccc4d208b18dd145d3805e7"
format_stamp: "Formatted at 2026-09-26 03:45:18 on dist-test-slave-n2th"
server_key: "2c616df07164c9ed0a19ddfdc0bab2de"
server_key_iv: "61843e3dab458d20dd5eb2e6d1c3ad03"
server_key_version: "encryptionkey@0"
I20260926 03:45:18.243640 11311 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790394221564982-779-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790394221564982-779-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790394221564982-779-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:45:18.271785 11311 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:45:18.276247 11311 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:45:18.276495 11311 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:45:18.277406 11311 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:45:18.278707 11311 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:45:18.278801 11311 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:45:18.278888 11311 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:45:18.278934 11311 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:45:18.312371 11311 rpc_server.cc:307] RPC server started. Bound to: 127.0.194.194:46025
I20260926 03:45:18.312407 11438 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.194.194:46025 every 8 connection(s)
I20260926 03:45:18.313868 11311 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790394221564982-779-0/minicluster-data/ts-1/data/info.pb
I20260926 03:45:18.321539 779 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu as pid 11311
I20260926 03:45:18.321779 779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790394221564982-779-0/minicluster-data/ts-1/wal/instance
I20260926 03:45:18.322201 779 external_mini_cluster.cc:1428] Setting key 064b47da5b4ee3c72033f7d7ea9098f4
Sep 26 03:45:18 dist-test-slave-n2th krb5kdc[11071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394318, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.194@KRBTEST.COM for kudu/127.0.194.254@KRBTEST.COM
I20260926 03:45:18.336153 11441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:45:18.316522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.194:42719 (local address 127.0.194.254:41957)
0926 03:45:18.316931 (+ 409us) server_negotiation.cc:206] Beginning negotiation
0926 03:45:18.316935 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:45:18.317946 (+ 1011us) server_negotiation.cc:407] Connection header received
0926 03:45:18.319824 (+ 1878us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:45:18.319830 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:45:18.319922 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:45:18.320041 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:45:18.321529 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:45:18.322410 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:45:18.324334 (+ 1924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:45:18.324757 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:45:18.324887 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:45:18.329672 (+ 4785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:45:18.329680 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:45:18.329688 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:45:18.329731 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:45:18.332945 (+ 3214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:45:18.333784 (+ 839us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:45:18.333788 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:45:18.333791 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:45:18.333917 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:45:18.334378 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:45:18.334383 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:45:18.334385 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:45:18.334654 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:45:18.334830 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0926 03:45:18.335548 (+ 718us) server_negotiation.cc:299] Negotiation successful
0926 03:45:18.335772 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":209,"threads_started":1}
I20260926 03:45:18.337329 11442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:45:18.317083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:41957 (local address 127.0.194.194:42719)
0926 03:45:18.317765 (+ 682us) negotiation.cc:107] Waiting for socket to connect
0926 03:45:18.317810 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 03:45:18.319550 (+ 1740us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:45:18.320280 (+ 730us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:45:18.320288 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:45:18.320795 (+ 507us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:45:18.321358 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:45:18.321376 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:45:18.322550 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:45:18.322560 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:45:18.324180 (+ 1620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:45:18.324192 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:45:18.324990 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:45:18.324997 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:45:18.325325 (+ 328us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:45:18.326132 (+ 807us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:45:18.326164 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:45:18.329406 (+ 3242us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:45:18.333098 (+ 3692us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:45:18.333106 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:45:18.333122 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:45:18.333617 (+ 495us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:45:18.334034 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:45:18.334041 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:45:18.334047 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:45:18.334239 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:45:18.334805 (+ 566us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:45:18.334814 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:45:18.335180 (+ 366us) client_negotiation.cc:769] Sending connection context
0926 03:45:18.335449 (+ 269us) client_negotiation.cc:240] Negotiation successful
0926 03:45:18.336012 (+ 563us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":265,"threads_started":1}
I20260926 03:45:18.338858 11439 heartbeater.cc:347] Connected to a master server at 127.0.194.254:41957
I20260926 03:45:18.339191 11439 heartbeater.cc:464] Registering TS with master...
I20260926 03:45:18.339917 11439 heartbeater.cc:511] Master 127.0.194.254:41957 requested a full tablet report, sending...
I20260926 03:45:18.341450 11116 ts_manager.cc:194] Registered new tserver with Master: 0adabfbeaccc4d208b18dd145d3805e7 (127.0.194.194:46025)
I20260926 03:45:18.342785 11116 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.194.194@KRBTEST.COM'} at 127.0.194.194:42719
WARNING: no policy specified for kudu/127.0.194.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.194.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.194.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790394221564982-779-0/krb5kdc/kudu_127.0.194.195.keytab.
Entry for principal kudu/127.0.194.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790394221564982-779-0/krb5kdc/kudu_127.0.194.195.keytab.
I20260926 03:45:18.375537 779 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
/tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790394221564982-779-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790394221564982-779-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790394221564982-779-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790394221564982-779-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.194.195:0
--local_ip_for_outbound_sockets=127.0.194.195
--webserver_interface=127.0.194.195
--webserver_port=0
--tserver_master_addrs=127.0.194.254:41957
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790394221564982-779-0/krb5kdc/kudu_127.0.194.195.keytab
--principal=kudu/127.0.194.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.194.212:37267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790394221564982-779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790394221564982-779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790394221564982-779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:45:18.706611 11445 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:45:18.707031 11445 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:45:18.707213 11445 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:45:18.714491 11445 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:45:18.714605 11445 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:45:18.714718 11445 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.194.195
I20260926 03:45:18.724623 11445 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.194.212:37267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790394221564982-779-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790394221564982-779-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790394221564982-779-0/krb5kdc/kudu_127.0.194.195.keytab
--principal=kudu/127.0.194.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.194.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790394221564982-779-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.194.195
--webserver_port=0
--tserver_master_addrs=127.0.194.254:41957
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.194.195
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790394221564982-779-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:45:18.725888 11445 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:45:18.727383 11445 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:45:18.740557 11450 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:45:18.740608 11453 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:45:18.740634 11451 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:45:18.741525 11445 server_base.cc:1034] running on GCE node
I20260926 03:45:18.742225 11445 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:45:18.743345 11445 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:45:18.744585 11445 hybrid_clock.cc:648] HybridClock initialized: now 1790394318744532 us; error 51 us; skew 500 ppm
I20260926 03:45:18.744923 11445 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:45:18 dist-test-slave-n2th krb5kdc[11071](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394318, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:45:18.748447 11445 init.cc:377] Logged in from keytab as kudu/127.0.194.195@KRBTEST.COM (short username kudu)
I20260926 03:45:18.749927 11445 webserver.cc:460] Webserver started at http://127.0.194.195:34355/ using document root <none> and password file <none>
I20260926 03:45:18.750658 11445 fs_manager.cc:359] Metadata directory not provided
I20260926 03:45:18.750815 11445 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:45:18.751140 11445 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:45:18.754122 11445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790394221564982-779-0/minicluster-data/ts-2/data/instance:
uuid: "4ad4c5a82c73413e86da8aa6b2f1ed4d"
format_stamp: "Formatted at 2026-09-26 03:45:18 on dist-test-slave-n2th"
server_key: "6bdf05e68c4dcb2a4fcb69fa45a3b19d"
server_key_iv: "1e2b8efe09f6c2fb300c802b7568bf1e"
server_key_version: "encryptionkey@0"
I20260926 03:45:18.754984 11445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790394221564982-779-0/minicluster-data/ts-2/wal/instance:
uuid: "4ad4c5a82c73413e86da8aa6b2f1ed4d"
format_stamp: "Formatted at 2026-09-26 03:45:18 on dist-test-slave-n2th"
server_key: "6bdf05e68c4dcb2a4fcb69fa45a3b19d"
server_key_iv: "1e2b8efe09f6c2fb300c802b7568bf1e"
server_key_version: "encryptionkey@0"
I20260926 03:45:18.760430 11445 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 03:45:18.764176 11460 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:45:18.765645 11445 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 03:45:18.765846 11445 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790394221564982-779-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790394221564982-779-0/minicluster-data/ts-2/wal
uuid: "4ad4c5a82c73413e86da8aa6b2f1ed4d"
format_stamp: "Formatted at 2026-09-26 03:45:18 on dist-test-slave-n2th"
server_key: "6bdf05e68c4dcb2a4fcb69fa45a3b19d"
server_key_iv: "1e2b8efe09f6c2fb300c802b7568bf1e"
server_key_version: "encryptionkey@0"
I20260926 03:45:18.766037 11445 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790394221564982-779-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790394221564982-779-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790394221564982-779-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:45:18.785805 11445 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:45:18.790846 11445 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:45:18.791122 11445 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:45:18.792078 11445 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:45:18.793464 11445 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:45:18.793538 11445 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:45:18.793633 11445 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:45:18.793678 11445 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:45:18.821661 11305 heartbeater.cc:503] Master 127.0.194.254:41957 was elected leader, sending a full tablet report...
I20260926 03:45:18.827629 11445 rpc_server.cc:307] RPC server started. Bound to: 127.0.194.195:44009
I20260926 03:45:18.827724 11572 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.194.195:44009 every 8 connection(s)
I20260926 03:45:18.828989 11445 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790394221564982-779-0/minicluster-data/ts-2/data/info.pb
I20260926 03:45:18.839212 779 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu as pid 11445
I20260926 03:45:18.839426 779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790394221564982-779-0/minicluster-data/ts-2/wal/instance
I20260926 03:45:18.839980 779 external_mini_cluster.cc:1428] Setting key 41f52fcca667e10065e143d06f899bb7
Sep 26 03:45:18 dist-test-slave-n2th krb5kdc[11071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394318, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.195@KRBTEST.COM for kudu/127.0.194.254@KRBTEST.COM
I20260926 03:45:18.852144 11441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:45:18.833162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.195:59891 (local address 127.0.194.254:41957)
0926 03:45:18.833411 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0926 03:45:18.833418 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:45:18.833668 (+ 250us) server_negotiation.cc:407] Connection header received
0926 03:45:18.835663 (+ 1995us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:45:18.835669 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:45:18.835767 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:45:18.835877 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:45:18.837516 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:45:18.838351 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:45:18.840764 (+ 2413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:45:18.841366 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:45:18.841499 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:45:18.845609 (+ 4110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:45:18.845614 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:45:18.845617 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:45:18.845652 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:45:18.848966 (+ 3314us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:45:18.849840 (+ 874us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:45:18.849844 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:45:18.849847 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:45:18.849974 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:45:18.850394 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:45:18.850397 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:45:18.850399 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:45:18.850663 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:45:18.850885 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0926 03:45:18.851676 (+ 791us) server_negotiation.cc:299] Negotiation successful
0926 03:45:18.851875 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260926 03:45:18.853217 11575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:45:18.832500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:41957 (local address 127.0.194.195:59891)
0926 03:45:18.833445 (+ 945us) negotiation.cc:107] Waiting for socket to connect
0926 03:45:18.833531 (+ 86us) client_negotiation.cc:174] Beginning negotiation
0926 03:45:18.835342 (+ 1811us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:45:18.836139 (+ 797us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:45:18.836148 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:45:18.836735 (+ 587us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:45:18.837340 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:45:18.837357 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:45:18.838511 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:45:18.838515 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:45:18.840605 (+ 2090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:45:18.840618 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:45:18.841623 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:45:18.841631 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:45:18.841953 (+ 322us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:45:18.842809 (+ 856us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:45:18.842849 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:45:18.845366 (+ 2517us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:45:18.849173 (+ 3807us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:45:18.849181 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:45:18.849197 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:45:18.849715 (+ 518us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:45:18.850082 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:45:18.850085 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:45:18.850087 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:45:18.850282 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:45:18.850846 (+ 564us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:45:18.850853 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:45:18.851235 (+ 382us) client_negotiation.cc:769] Sending connection context
0926 03:45:18.851505 (+ 270us) client_negotiation.cc:240] Negotiation successful
0926 03:45:18.851942 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":566,"thread_start_us":358,"threads_started":1}
I20260926 03:45:18.855315 11573 heartbeater.cc:347] Connected to a master server at 127.0.194.254:41957
I20260926 03:45:18.855683 11573 heartbeater.cc:464] Registering TS with master...
I20260926 03:45:18.856475 11573 heartbeater.cc:511] Master 127.0.194.254:41957 requested a full tablet report, sending...
I20260926 03:45:18.857962 11115 ts_manager.cc:194] Registered new tserver with Master: 4ad4c5a82c73413e86da8aa6b2f1ed4d (127.0.194.195:44009)
I20260926 03:45:18.858882 11115 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.194.195@KRBTEST.COM'} at 127.0.194.195:59891
I20260926 03:45:18.869167 779 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 03:45:18.887018 11441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:45:18.874049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49710 (local address 127.0.194.254:41957)
0926 03:45:18.874311 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0926 03:45:18.874320 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:45:18.874721 (+ 401us) server_negotiation.cc:407] Connection header received
0926 03:45:18.874933 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:45:18.874938 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:45:18.875024 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:45:18.875118 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:45:18.876601 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:45:18.877366 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:45:18.878610 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:45:18.879102 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:45:18.879204 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:45:18.881457 (+ 2253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:45:18.881462 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:45:18.881466 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:45:18.881528 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:45:18.884349 (+ 2821us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:45:18.885055 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:45:18.885060 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:45:18.885063 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:45:18.885212 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:45:18.885585 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:45:18.885589 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:45:18.885591 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:45:18.885920 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:45:18.886126 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0926 03:45:18.886405 (+ 279us) server_negotiation.cc:299] Negotiation successful
0926 03:45:18.886585 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260926 03:45:18.891372 11115 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49710:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 03:45:18.898454 11115 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 03:45:18.922132 11586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:45:18.912247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34790 (local address 127.0.194.193:41163)
0926 03:45:18.912862 (+ 615us) server_negotiation.cc:206] Beginning negotiation
0926 03:45:18.912869 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:45:18.913142 (+ 273us) server_negotiation.cc:407] Connection header received
0926 03:45:18.913452 (+ 310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:45:18.913461 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:45:18.913764 (+ 303us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:45:18.913973 (+ 209us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:45:18.914869 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:45:18.916472 (+ 1603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:45:18.919253 (+ 2781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:45:18.920848 (+ 1595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:45:18.921002 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:45:18.921239 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0926 03:45:18.921593 (+ 354us) server_negotiation.cc:299] Negotiation successful
0926 03:45:18.921766 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":460,"thread_start_us":332,"threads_started":1}
I20260926 03:45:18.922197 11585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:45:18.911991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.193:41163 (local address 127.0.0.1:34790)
0926 03:45:18.912997 (+ 1006us) negotiation.cc:107] Waiting for socket to connect
0926 03:45:18.913033 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 03:45:18.913225 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:45:18.914265 (+ 1040us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:45:18.914270 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:45:18.914299 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:45:18.914628 (+ 329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:45:18.914642 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:45:18.916658 (+ 2016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:45:18.916663 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:45:18.919055 (+ 2392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:45:18.919070 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:45:18.921083 (+ 2013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:45:18.921091 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:45:18.921311 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:45:18.921360 (+ 49us) client_negotiation.cc:769] Sending connection context
0926 03:45:18.921521 (+ 161us) client_negotiation.cc:240] Negotiation successful
0926 03:45:18.921756 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":786,"mutex_wait_us":4,"thread_start_us":648,"threads_started":1}
I20260926 03:45:18.924983 11587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:45:18.913451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.195:44009 (local address 127.0.0.1:34422)
0926 03:45:18.913976 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0926 03:45:18.913995 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 03:45:18.914099 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:45:18.917320 (+ 3221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:45:18.917325 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:45:18.917359 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:45:18.917906 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:45:18.917920 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:45:18.919650 (+ 1730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:45:18.919665 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:45:18.921646 (+ 1981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:45:18.921658 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:45:18.924020 (+ 2362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:45:18.924029 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:45:18.924273 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:45:18.924314 (+ 41us) client_negotiation.cc:769] Sending connection context
0926 03:45:18.924410 (+ 96us) client_negotiation.cc:240] Negotiation successful
0926 03:45:18.924580 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":411,"mutex_wait_us":34,"thread_start_us":276,"threads_started":1}
I20260926 03:45:18.925007 11588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:45:18.913832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34422 (local address 127.0.194.195:44009)
0926 03:45:18.916633 (+ 2801us) server_negotiation.cc:206] Beginning negotiation
0926 03:45:18.916644 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 03:45:18.916682 (+ 38us) server_negotiation.cc:407] Connection header received
0926 03:45:18.916799 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:45:18.916807 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:45:18.917101 (+ 294us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:45:18.917298 (+ 197us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:45:18.918129 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:45:18.919493 (+ 1364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:45:18.921814 (+ 2321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:45:18.923337 (+ 1523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:45:18.923466 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:45:18.923839 (+ 373us) server_negotiation.cc:1035] Waiting for connection context
0926 03:45:18.924484 (+ 645us) server_negotiation.cc:299] Negotiation successful
0926 03:45:18.924659 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2613,"thread_start_us":2446,"threads_started":1}
I20260926 03:45:18.925097 11589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:45:18.914360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55478 (local address 127.0.194.194:46025)
0926 03:45:18.914957 (+ 597us) server_negotiation.cc:206] Beginning negotiation
0926 03:45:18.914969 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 03:45:18.915836 (+ 867us) server_negotiation.cc:407] Connection header received
0926 03:45:18.916809 (+ 973us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:45:18.916815 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:45:18.917095 (+ 280us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:45:18.917286 (+ 191us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:45:18.918223 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:45:18.919757 (+ 1534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:45:18.922189 (+ 2432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:45:18.923794 (+ 1605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:45:18.923919 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:45:18.924153 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0926 03:45:18.924613 (+ 460us) server_negotiation.cc:299] Negotiation successful
0926 03:45:18.924795 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":431,"thread_start_us":271,"threads_started":1}
I20260926 03:45:18.925859 11590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:45:18.915021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.194:46025 (local address 127.0.0.1:55478)
0926 03:45:18.915717 (+ 696us) negotiation.cc:107] Waiting for socket to connect
0926 03:45:18.915736 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 03:45:18.916581 (+ 845us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:45:18.917591 (+ 1010us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:45:18.917594 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:45:18.917615 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:45:18.917989 (+ 374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:45:18.917997 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:45:18.919919 (+ 1922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:45:18.919926 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:45:18.921979 (+ 2053us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:45:18.921992 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:45:18.924214 (+ 2222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:45:18.924220 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:45:18.924425 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:45:18.924451 (+ 26us) client_negotiation.cc:769] Sending connection context
0926 03:45:18.925298 (+ 847us) client_negotiation.cc:240] Negotiation successful
0926 03:45:18.925489 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":444,"mutex_wait_us":9,"thread_start_us":278,"threads_started":1}
I20260926 03:45:18.927282 11240 tablet_service.cc:1467] Processing CreateTablet for tablet ab90fb7f22ae4bcbb3e54e6b67cc57c4 (DEFAULT_TABLE table=test-table [id=1e3be51ce7f44be3b65170f8d2d8f487]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:45:18.928974 11240 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab90fb7f22ae4bcbb3e54e6b67cc57c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:45:18.929229 11508 tablet_service.cc:1467] Processing CreateTablet for tablet ab90fb7f22ae4bcbb3e54e6b67cc57c4 (DEFAULT_TABLE table=test-table [id=1e3be51ce7f44be3b65170f8d2d8f487]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:45:18.930400 11508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab90fb7f22ae4bcbb3e54e6b67cc57c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:45:18.929972 11374 tablet_service.cc:1467] Processing CreateTablet for tablet ab90fb7f22ae4bcbb3e54e6b67cc57c4 (DEFAULT_TABLE table=test-table [id=1e3be51ce7f44be3b65170f8d2d8f487]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:45:18.931605 11374 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab90fb7f22ae4bcbb3e54e6b67cc57c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:45:18.943107 11591 tablet_bootstrap.cc:492] T ab90fb7f22ae4bcbb3e54e6b67cc57c4 P 4ad4c5a82c73413e86da8aa6b2f1ed4d: Bootstrap starting.
I20260926 03:45:18.947422 11591 tablet_bootstrap.cc:654] T ab90fb7f22ae4bcbb3e54e6b67cc57c4 P 4ad4c5a82c73413e86da8aa6b2f1ed4d: Neither blocks nor log segments found. Creating new log.
I20260926 03:45:18.948529 11593 tablet_bootstrap.cc:492] T ab90fb7f22ae4bcbb3e54e6b67cc57c4 P 0adabfbeaccc4d208b18dd145d3805e7: Bootstrap starting.
I20260926 03:45:18.948975 11591 log.cc:824] T ab90fb7f22ae4bcbb3e54e6b67cc57c4 P 4ad4c5a82c73413e86da8aa6b2f1ed4d: Log is configured to *not* fsync() on all Append() calls
I20260926 03:45:18.950899 11592 tablet_bootstrap.cc:492] T ab90fb7f22ae4bcbb3e54e6b67cc57c4 P c35c4f35e08e4b3dbf9a9c25275ed908: Bootstrap starting.
I20260926 03:45:18.952572 11593 tablet_bootstrap.cc:654] T ab90fb7f22ae4bcbb3e54e6b67cc57c4 P 0adabfbeaccc4d208b18dd145d3805e7: Neither blocks nor log segments found. Creating new log.
I20260926 03:45:18.953286 11591 tablet_bootstrap.cc:492] T ab90fb7f22ae4bcbb3e54e6b67cc57c4 P 4ad4c5a82c73413e86da8aa6b2f1ed4d: No bootstrap required, opened a new log
I20260926 03:45:18.953593 11591 ts_tablet_manager.cc:1397] T ab90fb7f22ae4bcbb3e54e6b67cc57c4 P 4ad4c5a82c73413e86da8aa6b2f1ed4d: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260926 03:45:18.954269 11593 log.cc:824] T ab90fb7f22ae4bcbb3e54e6b67cc57c4 P 0adabfbeaccc4d208b18dd145d3805e7: Log is configured to *not* fsync() on all Append() calls
I20260926 03:45:18.954916 11592 tablet_bootstrap.cc:654] T ab90fb7f22ae4bcbb3e54e6b67cc57c4 P c35c4f35e08e4b3dbf9a9c25275ed908: Neither blocks nor log segments found. Creating new log.
I20260926 03:45:18.956163 11592 log.cc:824] T ab90fb7f22ae4bcbb3e54e6b67cc57c4 P c35c4f35e08e4b3dbf9a9c25275ed908: Log is configured to *not* fsync() on all Append() calls
I20260926 03:45:18.957252 11593 tablet_bootstrap.cc:492] T ab90fb7f22ae4bcbb3e54e6b67cc57c4 P 0adabfbeaccc4d208b18dd145d3805e7: No bootstrap required, opened a new log
I20260926 03:45:18.957563 11593 ts_tablet_manager.cc:1397] T ab90fb7f22ae4bcbb3e54e6b67cc57c4 P 0adabfbeaccc4d208b18dd145d3805e7: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.002s
I20260926 03:45:18.959331 11592 tablet_bootstrap.cc:492] T ab90fb7f22ae4bcbb3e54e6b67cc57c4 P c35c4f35e08e4b3dbf9a9c25275ed908: No bootstrap required, opened a new log
I20260926 03:45:18.959602 11592 ts_tablet_manager.cc:1397] T ab90fb7f22ae4bcbb3e54e6b67cc57c4 P c35c4f35e08e4b3dbf9a9c25275ed908: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.005s
I20260926 03:45:18.964375 11591 raft_consensus.cc:348] T ab90fb7f22ae4bcbb3e54e6b67cc57c4 P 4ad4c5a82c73413e86da8aa6b2f1ed4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c35c4f35e08e4b3dbf9a9c25275ed908" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 41163 } } peers { permanent_uuid: "4ad4c5a82c73413e86da8aa6b2f1ed4d" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 44009 } } peers { permanent_uuid: "0adabfbeaccc4d208b18dd145d3805e7" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 46025 } }
I20260926 03:45:18.964888 11591 raft_consensus.cc:374] T ab90fb7f22ae4bcbb3e54e6b67cc57c4 P 4ad4c5a82c73413e86da8aa6b2f1ed4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:45:18.964998 11591 raft_consensus.cc:729] T ab90fb7f22ae4bcbb3e54e6b67cc57c4 P 4ad4c5a82c73413e86da8aa6b2f1ed4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ad4c5a82c73413e86da8aa6b2f1ed4d, State: Initialized, Role: FOLLOWER
I20260926 03:45:18.965281 11593 raft_consensus.cc:348] T ab90fb7f22ae4bcbb3e54e6b67cc57c4 P 0adabfbeaccc4d208b18dd145d3805e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c35c4f35e08e4b3dbf9a9c25275ed908" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 41163 } } peers { permanent_uuid: "4ad4c5a82c73413e86da8aa6b2f1ed4d" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 44009 } } peers { permanent_uuid: "0adabfbeaccc4d208b18dd145d3805e7" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 46025 } }
I20260926 03:45:18.965731 11593 raft_consensus.cc:374] T ab90fb7f22ae4bcbb3e54e6b67cc57c4 P 0adabfbeaccc4d208b18dd145d3805e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:45:18.965839 11593 raft_consensus.cc:729] T ab90fb7f22ae4bcbb3e54e6b67cc57c4 P 0adabfbeaccc4d208b18dd145d3805e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0adabfbeaccc4d208b18dd145d3805e7, State: Initialized, Role: FOLLOWER
I20260926 03:45:18.965699 11591 consensus_queue.cc:260] T ab90fb7f22ae4bcbb3e54e6b67cc57c4 P 4ad4c5a82c73413e86da8aa6b2f1ed4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c35c4f35e08e4b3dbf9a9c25275ed908" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 41163 } } peers { permanent_uuid: "4ad4c5a82c73413e86da8aa6b2f1ed4d" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 44009 } } peers { permanent_uuid: "0adabfbeaccc4d208b18dd145d3805e7" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 46025 } }
I20260926 03:45:18.966446 11593 consensus_queue.cc:260] T ab90fb7f22ae4bcbb3e54e6b67cc57c4 P 0adabfbeaccc4d208b18dd145d3805e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c35c4f35e08e4b3dbf9a9c25275ed908" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 41163 } } peers { permanent_uuid: "4ad4c5a82c73413e86da8aa6b2f1ed4d" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 44009 } } peers { permanent_uuid: "0adabfbeaccc4d208b18dd145d3805e7" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 46025 } }
I20260926 03:45:18.967599 11439 heartbeater.cc:503] Master 127.0.194.254:41957 was elected leader, sending a full tablet report...
I20260926 03:45:18.968344 11593 ts_tablet_manager.cc:1428] T ab90fb7f22ae4bcbb3e54e6b67cc57c4 P 0adabfbeaccc4d208b18dd145d3805e7: Time spent starting tablet: real 0.011s user 0.006s sys 0.005s
I20260926 03:45:18.971612 11573 heartbeater.cc:503] Master 127.0.194.254:41957 was elected leader, sending a full tablet report...
I20260926 03:45:18.972203 11591 ts_tablet_manager.cc:1428] T ab90fb7f22ae4bcbb3e54e6b67cc57c4 P 4ad4c5a82c73413e86da8aa6b2f1ed4d: Time spent starting tablet: real 0.018s user 0.012s sys 0.008s
I20260926 03:45:18.971839 11592 raft_consensus.cc:348] T ab90fb7f22ae4bcbb3e54e6b67cc57c4 P c35c4f35e08e4b3dbf9a9c25275ed908 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c35c4f35e08e4b3dbf9a9c25275ed908" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 41163 } } peers { permanent_uuid: "4ad4c5a82c73413e86da8aa6b2f1ed4d" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 44009 } } peers { permanent_uuid: "0adabfbeaccc4d208b18dd145d3805e7" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 46025 } }
I20260926 03:45:18.972401 11592 raft_consensus.cc:374] T ab90fb7f22ae4bcbb3e54e6b67cc57c4 P c35c4f35e08e4b3dbf9a9c25275ed908 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:45:18.972491 11592 raft_consensus.cc:729] T ab90fb7f22ae4bcbb3e54e6b67cc57c4 P c35c4f35e08e4b3dbf9a9c25275ed908 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c35c4f35e08e4b3dbf9a9c25275ed908, State: Initialized, Role: FOLLOWER
I20260926 03:45:18.973237 11592 consensus_queue.cc:260] T ab90fb7f22ae4bcbb3e54e6b67cc57c4 P c35c4f35e08e4b3dbf9a9c25275ed908 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c35c4f35e08e4b3dbf9a9c25275ed908" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 41163 } } peers { permanent_uuid: "4ad4c5a82c73413e86da8aa6b2f1ed4d" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 44009 } } peers { permanent_uuid: "0adabfbeaccc4d208b18dd145d3805e7" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 46025 } }
I20260926 03:45:18.975162 11592 ts_tablet_manager.cc:1428] T ab90fb7f22ae4bcbb3e54e6b67cc57c4 P c35c4f35e08e4b3dbf9a9c25275ed908: Time spent starting tablet: real 0.015s user 0.012s sys 0.004s
W20260926 03:45:19.039469 11306 tablet.cc:2369] T ab90fb7f22ae4bcbb3e54e6b67cc57c4 P c35c4f35e08e4b3dbf9a9c25275ed908: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 03:45:19.066167 11440 tablet.cc:2369] T ab90fb7f22ae4bcbb3e54e6b67cc57c4 P 0adabfbeaccc4d208b18dd145d3805e7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 03:45:19.079537 11599 raft_consensus.cc:482] T ab90fb7f22ae4bcbb3e54e6b67cc57c4 P c35c4f35e08e4b3dbf9a9c25275ed908 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 03:45:19.079851 11599 raft_consensus.cc:504] T ab90fb7f22ae4bcbb3e54e6b67cc57c4 P c35c4f35e08e4b3dbf9a9c25275ed908 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c35c4f35e08e4b3dbf9a9c25275ed908" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 41163 } } peers { permanent_uuid: "4ad4c5a82c73413e86da8aa6b2f1ed4d" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 44009 } } peers { permanent_uuid: "0adabfbeaccc4d208b18dd145d3805e7" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 46025 } }
W20260926 03:45:19.080596 11574 tablet.cc:2369] T ab90fb7f22ae4bcbb3e54e6b67cc57c4 P 4ad4c5a82c73413e86da8aa6b2f1ed4d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 03:45:19.081552 11599 leader_election.cc:288] T ab90fb7f22ae4bcbb3e54e6b67cc57c4 P c35c4f35e08e4b3dbf9a9c25275ed908 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4ad4c5a82c73413e86da8aa6b2f1ed4d (127.0.194.195:44009), 0adabfbeaccc4d208b18dd145d3805e7 (127.0.194.194:46025)
I20260926 03:45:19.088454 11588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:45:19.081986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.193:45417 (local address 127.0.194.195:44009)
0926 03:45:19.082165 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0926 03:45:19.082170 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:45:19.082765 (+ 595us) server_negotiation.cc:407] Connection header received
0926 03:45:19.083019 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:45:19.083024 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:45:19.083105 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:45:19.083213 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:45:19.083804 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:45:19.084536 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:45:19.086566 (+ 2030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:45:19.087401 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:45:19.087621 (+ 220us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:45:19.087704 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 03:45:19.088052 (+ 348us) server_negotiation.cc:299] Negotiation successful
0926 03:45:19.088182 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 03:45:19.088611 11600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:45:19.081969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.195:44009 (local address 127.0.194.193:45417)
0926 03:45:19.082634 (+ 665us) negotiation.cc:107] Waiting for socket to connect
0926 03:45:19.082655 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 03:45:19.082842 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:45:19.083285 (+ 443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:45:19.083290 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:45:19.083322 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:45:19.083657 (+ 335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:45:19.083666 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:45:19.084685 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:45:19.084693 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:45:19.086390 (+ 1697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:45:19.086402 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:45:19.087661 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:45:19.087667 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:45:19.087850 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:45:19.087874 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 03:45:19.088012 (+ 138us) client_negotiation.cc:240] Negotiation successful
0926 03:45:19.088182 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":540,"mutex_wait_us":1,"thread_start_us":357,"threads_started":1}
I20260926 03:45:19.088747 11589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:45:19.083137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.193:50051 (local address 127.0.194.194:46025)
0926 03:45:19.083472 (+ 335us) server_negotiation.cc:206] Beginning negotiation
0926 03:45:19.083480 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:45:19.083569 (+ 89us) server_negotiation.cc:407] Connection header received
0926 03:45:19.083768 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:45:19.083772 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:45:19.083826 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:45:19.083916 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:45:19.084423 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:45:19.085095 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:45:19.086917 (+ 1822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:45:19.087759 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:45:19.087951 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:45:19.088025 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 03:45:19.088324 (+ 299us) server_negotiation.cc:299] Negotiation successful
0926 03:45:19.088463 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153}
I20260926 03:45:19.088918 11601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:45:19.082804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.194:46025 (local address 127.0.194.193:50051)
0926 03:45:19.083460 (+ 656us) negotiation.cc:107] Waiting for socket to connect
0926 03:45:19.083482 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 03:45:19.083642 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:45:19.083949 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:45:19.083952 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:45:19.083978 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:45:19.084285 (+ 307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:45:19.084293 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:45:19.085232 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:45:19.085238 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:45:19.086772 (+ 1534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:45:19.086792 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:45:19.087989 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:45:19.087996 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:45:19.088174 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:45:19.088196 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 03:45:19.088297 (+ 101us) client_negotiation.cc:240] Negotiation successful
0926 03:45:19.088428 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":529,"mutex_wait_us":60,"thread_start_us":339,"threads_started":1}
I20260926 03:45:19.089910 11394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab90fb7f22ae4bcbb3e54e6b67cc57c4" candidate_uuid: "c35c4f35e08e4b3dbf9a9c25275ed908" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0adabfbeaccc4d208b18dd145d3805e7" is_pre_election: true
I20260926 03:45:19.089903 11528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab90fb7f22ae4bcbb3e54e6b67cc57c4" candidate_uuid: "c35c4f35e08e4b3dbf9a9c25275ed908" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ad4c5a82c73413e86da8aa6b2f1ed4d" is_pre_election: true
I20260926 03:45:19.090451 11394 raft_consensus.cc:2454] T ab90fb7f22ae4bcbb3e54e6b67cc57c4 P 0adabfbeaccc4d208b18dd145d3805e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c35c4f35e08e4b3dbf9a9c25275ed908 in term 0.
I20260926 03:45:19.090471 11528 raft_consensus.cc:2454] T ab90fb7f22ae4bcbb3e54e6b67cc57c4 P 4ad4c5a82c73413e86da8aa6b2f1ed4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c35c4f35e08e4b3dbf9a9c25275ed908 in term 0.
I20260926 03:45:19.091212 11196 leader_election.cc:302] T ab90fb7f22ae4bcbb3e54e6b67cc57c4 P c35c4f35e08e4b3dbf9a9c25275ed908 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0adabfbeaccc4d208b18dd145d3805e7, c35c4f35e08e4b3dbf9a9c25275ed908; no voters:
I20260926 03:45:19.091612 11599 raft_consensus.cc:2781] T ab90fb7f22ae4bcbb3e54e6b67cc57c4 P c35c4f35e08e4b3dbf9a9c25275ed908 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 03:45:19.091754 11599 raft_consensus.cc:482] T ab90fb7f22ae4bcbb3e54e6b67cc57c4 P c35c4f35e08e4b3dbf9a9c25275ed908 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 03:45:19.091853 11599 raft_consensus.cc:3037] T ab90fb7f22ae4bcbb3e54e6b67cc57c4 P c35c4f35e08e4b3dbf9a9c25275ed908 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:45:19.094072 11599 raft_consensus.cc:504] T ab90fb7f22ae4bcbb3e54e6b67cc57c4 P c35c4f35e08e4b3dbf9a9c25275ed908 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c35c4f35e08e4b3dbf9a9c25275ed908" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 41163 } } peers { permanent_uuid: "4ad4c5a82c73413e86da8aa6b2f1ed4d" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 44009 } } peers { permanent_uuid: "0adabfbeaccc4d208b18dd145d3805e7" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 46025 } }
I20260926 03:45:19.094705 11599 leader_election.cc:288] T ab90fb7f22ae4bcbb3e54e6b67cc57c4 P c35c4f35e08e4b3dbf9a9c25275ed908 [CANDIDATE]: Term 1 election: Requested vote from peers 4ad4c5a82c73413e86da8aa6b2f1ed4d (127.0.194.195:44009), 0adabfbeaccc4d208b18dd145d3805e7 (127.0.194.194:46025)
I20260926 03:45:19.095396 11528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab90fb7f22ae4bcbb3e54e6b67cc57c4" candidate_uuid: "c35c4f35e08e4b3dbf9a9c25275ed908" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ad4c5a82c73413e86da8aa6b2f1ed4d"
I20260926 03:45:19.095396 11394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab90fb7f22ae4bcbb3e54e6b67cc57c4" candidate_uuid: "c35c4f35e08e4b3dbf9a9c25275ed908" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0adabfbeaccc4d208b18dd145d3805e7"
I20260926 03:45:19.095646 11394 raft_consensus.cc:3037] T ab90fb7f22ae4bcbb3e54e6b67cc57c4 P 0adabfbeaccc4d208b18dd145d3805e7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:45:19.095646 11528 raft_consensus.cc:3037] T ab90fb7f22ae4bcbb3e54e6b67cc57c4 P 4ad4c5a82c73413e86da8aa6b2f1ed4d [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:45:19.098928 11394 raft_consensus.cc:2454] T ab90fb7f22ae4bcbb3e54e6b67cc57c4 P 0adabfbeaccc4d208b18dd145d3805e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c35c4f35e08e4b3dbf9a9c25275ed908 in term 1.
I20260926 03:45:19.098928 11528 raft_consensus.cc:2454] T ab90fb7f22ae4bcbb3e54e6b67cc57c4 P 4ad4c5a82c73413e86da8aa6b2f1ed4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c35c4f35e08e4b3dbf9a9c25275ed908 in term 1.
I20260926 03:45:19.099428 11196 leader_election.cc:302] T ab90fb7f22ae4bcbb3e54e6b67cc57c4 P c35c4f35e08e4b3dbf9a9c25275ed908 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ad4c5a82c73413e86da8aa6b2f1ed4d, c35c4f35e08e4b3dbf9a9c25275ed908; no voters:
I20260926 03:45:19.099781 11599 raft_consensus.cc:2781] T ab90fb7f22ae4bcbb3e54e6b67cc57c4 P c35c4f35e08e4b3dbf9a9c25275ed908 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:45:19.100262 11599 raft_consensus.cc:686] T ab90fb7f22ae4bcbb3e54e6b67cc57c4 P c35c4f35e08e4b3dbf9a9c25275ed908 [term 1 LEADER]: Becoming Leader. State: Replica: c35c4f35e08e4b3dbf9a9c25275ed908, State: Running, Role: LEADER
I20260926 03:45:19.100798 11599 consensus_queue.cc:237] T ab90fb7f22ae4bcbb3e54e6b67cc57c4 P c35c4f35e08e4b3dbf9a9c25275ed908 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c35c4f35e08e4b3dbf9a9c25275ed908" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 41163 } } peers { permanent_uuid: "4ad4c5a82c73413e86da8aa6b2f1ed4d" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 44009 } } peers { permanent_uuid: "0adabfbeaccc4d208b18dd145d3805e7" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 46025 } }
I20260926 03:45:19.105801 11116 catalog_manager.cc:5495] T ab90fb7f22ae4bcbb3e54e6b67cc57c4 P c35c4f35e08e4b3dbf9a9c25275ed908 reported cstate change: term changed from 0 to 1, leader changed from <none> to c35c4f35e08e4b3dbf9a9c25275ed908 (127.0.194.193). New cstate: current_term: 1 leader_uuid: "c35c4f35e08e4b3dbf9a9c25275ed908" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c35c4f35e08e4b3dbf9a9c25275ed908" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 41163 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4ad4c5a82c73413e86da8aa6b2f1ed4d" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 44009 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0adabfbeaccc4d208b18dd145d3805e7" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 46025 } health_report { overall_health: UNKNOWN } } }
I20260926 03:45:19.127602 11586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:45:19.122278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34798 (local address 127.0.194.193:41163)
0926 03:45:19.122503 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0926 03:45:19.122511 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:45:19.122610 (+ 99us) server_negotiation.cc:407] Connection header received
0926 03:45:19.122842 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:45:19.122846 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:45:19.122926 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:45:19.123053 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:45:19.123402 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:45:19.124115 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:45:19.125386 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:45:19.125798 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:45:19.125890 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:45:19.126371 (+ 481us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:45:19.127009 (+ 638us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:45:19.127130 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0926 03:45:19.127223 (+ 93us) server_negotiation.cc:299] Negotiation successful
0926 03:45:19.127287 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 03:45:19.136211 11528 raft_consensus.cc:1261] T ab90fb7f22ae4bcbb3e54e6b67cc57c4 P 4ad4c5a82c73413e86da8aa6b2f1ed4d [term 1 FOLLOWER]: Refusing update from remote peer c35c4f35e08e4b3dbf9a9c25275ed908: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:45:19.136211 11394 raft_consensus.cc:1261] T ab90fb7f22ae4bcbb3e54e6b67cc57c4 P 0adabfbeaccc4d208b18dd145d3805e7 [term 1 FOLLOWER]: Refusing update from remote peer c35c4f35e08e4b3dbf9a9c25275ed908: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:45:19.137269 11599 consensus_queue.cc:1035] T ab90fb7f22ae4bcbb3e54e6b67cc57c4 P c35c4f35e08e4b3dbf9a9c25275ed908 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0adabfbeaccc4d208b18dd145d3805e7" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 46025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:45:19.137602 11602 consensus_queue.cc:1035] T ab90fb7f22ae4bcbb3e54e6b67cc57c4 P c35c4f35e08e4b3dbf9a9c25275ed908 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ad4c5a82c73413e86da8aa6b2f1ed4d" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 44009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:45:19.152279 11606 mvcc.cc:205] Tried to move back new op lower bound from 7333455131170828288 to 7333455131040600064. Current Snapshot: MvccSnapshot[applied={T|T < 7333455131170828288}]
I20260926 03:45:19.152598 11605 mvcc.cc:205] Tried to move back new op lower bound from 7333455131170828288 to 7333455131040600064. Current Snapshot: MvccSnapshot[applied={T|T < 7333455131170828288}]
I20260926 03:45:19.172983 11116 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49710:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:45:19.173369 11116 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49710:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:45:19.178876 11116 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e34a4090276f4d659f9290a3494c90b3: Sending DeleteTablet for 3 replicas of tablet ab90fb7f22ae4bcbb3e54e6b67cc57c4
I20260926 03:45:19.180312 11240 tablet_service.cc:1514] Processing DeleteTablet for tablet ab90fb7f22ae4bcbb3e54e6b67cc57c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:45:19 UTC) from {username='kudu', principal='kudu/127.0.194.254@KRBTEST.COM'} at 127.0.0.1:34790
I20260926 03:45:19.182813 11508 tablet_service.cc:1514] Processing DeleteTablet for tablet ab90fb7f22ae4bcbb3e54e6b67cc57c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:45:19 UTC) from {username='kudu', principal='kudu/127.0.194.254@KRBTEST.COM'} at 127.0.0.1:34422
I20260926 03:45:19.183616 11374 tablet_service.cc:1514] Processing DeleteTablet for tablet ab90fb7f22ae4bcbb3e54e6b67cc57c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:45:19 UTC) from {username='kudu', principal='kudu/127.0.194.254@KRBTEST.COM'} at 127.0.0.1:55478
I20260926 03:45:19.189267 779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu pid 11177
I20260926 03:45:19.207657 11615 tablet_replica.cc:330] T ab90fb7f22ae4bcbb3e54e6b67cc57c4 P c35c4f35e08e4b3dbf9a9c25275ed908: stopping tablet replica
I20260926 03:45:19.208245 11615 raft_consensus.cc:2229] T ab90fb7f22ae4bcbb3e54e6b67cc57c4 P c35c4f35e08e4b3dbf9a9c25275ed908 [term 1 LEADER]: Raft consensus shutting down.
I20260926 03:45:19.209348 11616 tablet_replica.cc:330] T ab90fb7f22ae4bcbb3e54e6b67cc57c4 P 4ad4c5a82c73413e86da8aa6b2f1ed4d: stopping tablet replica
I20260926 03:45:19.209858 11616 raft_consensus.cc:2229] T ab90fb7f22ae4bcbb3e54e6b67cc57c4 P 4ad4c5a82c73413e86da8aa6b2f1ed4d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:45:19.210374 11616 raft_consensus.cc:2258] T ab90fb7f22ae4bcbb3e54e6b67cc57c4 P 4ad4c5a82c73413e86da8aa6b2f1ed4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:45:19.210660 11615 raft_consensus.cc:2258] T ab90fb7f22ae4bcbb3e54e6b67cc57c4 P c35c4f35e08e4b3dbf9a9c25275ed908 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:45:19.212991 11615 ts_tablet_manager.cc:1905] T ab90fb7f22ae4bcbb3e54e6b67cc57c4 P c35c4f35e08e4b3dbf9a9c25275ed908: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:45:19.214725 11617 tablet_replica.cc:330] T ab90fb7f22ae4bcbb3e54e6b67cc57c4 P 0adabfbeaccc4d208b18dd145d3805e7: stopping tablet replica
I20260926 03:45:19.215274 11617 raft_consensus.cc:2229] T ab90fb7f22ae4bcbb3e54e6b67cc57c4 P 0adabfbeaccc4d208b18dd145d3805e7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:45:19.215866 11617 raft_consensus.cc:2258] T ab90fb7f22ae4bcbb3e54e6b67cc57c4 P 0adabfbeaccc4d208b18dd145d3805e7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:45:19.220872 11616 ts_tablet_manager.cc:1905] T ab90fb7f22ae4bcbb3e54e6b67cc57c4 P 4ad4c5a82c73413e86da8aa6b2f1ed4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:45:19.221073 11615 ts_tablet_manager.cc:1918] T ab90fb7f22ae4bcbb3e54e6b67cc57c4 P c35c4f35e08e4b3dbf9a9c25275ed908: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:45:19.221256 11615 log.cc:1196] T ab90fb7f22ae4bcbb3e54e6b67cc57c4 P c35c4f35e08e4b3dbf9a9c25275ed908: Deleting WAL directory at /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790394221564982-779-0/minicluster-data/ts-0/wal/wals/ab90fb7f22ae4bcbb3e54e6b67cc57c4
I20260926 03:45:19.221859 11615 ts_tablet_manager.cc:1939] T ab90fb7f22ae4bcbb3e54e6b67cc57c4 P c35c4f35e08e4b3dbf9a9c25275ed908: Deleting consensus metadata
Sep 26 03:45:19 dist-test-slave-n2th krb5kdc[11071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394316, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.194.193@KRBTEST.COM
I20260926 03:45:19.223778 11103 catalog_manager.cc:4841] TS c35c4f35e08e4b3dbf9a9c25275ed908 (127.0.194.193:41163): tablet ab90fb7f22ae4bcbb3e54e6b67cc57c4 (table test-table [id=1e3be51ce7f44be3b65170f8d2d8f487]) successfully deleted
I20260926 03:45:19.224728 11617 ts_tablet_manager.cc:1905] T ab90fb7f22ae4bcbb3e54e6b67cc57c4 P 0adabfbeaccc4d208b18dd145d3805e7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:45:19.229542 11616 ts_tablet_manager.cc:1918] T ab90fb7f22ae4bcbb3e54e6b67cc57c4 P 4ad4c5a82c73413e86da8aa6b2f1ed4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:45:19.229802 11616 log.cc:1196] T ab90fb7f22ae4bcbb3e54e6b67cc57c4 P 4ad4c5a82c73413e86da8aa6b2f1ed4d: Deleting WAL directory at /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790394221564982-779-0/minicluster-data/ts-2/wal/wals/ab90fb7f22ae4bcbb3e54e6b67cc57c4
I20260926 03:45:19.230432 11616 ts_tablet_manager.cc:1939] T ab90fb7f22ae4bcbb3e54e6b67cc57c4 P 4ad4c5a82c73413e86da8aa6b2f1ed4d: Deleting consensus metadata
I20260926 03:45:19.232009 11617 ts_tablet_manager.cc:1918] T ab90fb7f22ae4bcbb3e54e6b67cc57c4 P 0adabfbeaccc4d208b18dd145d3805e7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:45:19.232178 11617 log.cc:1196] T ab90fb7f22ae4bcbb3e54e6b67cc57c4 P 0adabfbeaccc4d208b18dd145d3805e7: Deleting WAL directory at /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790394221564982-779-0/minicluster-data/ts-1/wal/wals/ab90fb7f22ae4bcbb3e54e6b67cc57c4
I20260926 03:45:19.232508 11103 catalog_manager.cc:4841] TS 4ad4c5a82c73413e86da8aa6b2f1ed4d (127.0.194.195:44009): tablet ab90fb7f22ae4bcbb3e54e6b67cc57c4 (table test-table [id=1e3be51ce7f44be3b65170f8d2d8f487]) successfully deleted
I20260926 03:45:19.232692 11617 ts_tablet_manager.cc:1939] T ab90fb7f22ae4bcbb3e54e6b67cc57c4 P 0adabfbeaccc4d208b18dd145d3805e7: Deleting consensus metadata
I20260926 03:45:19.232741 11586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:45:19.190356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34802 (local address 127.0.194.193:41163)
0926 03:45:19.190585 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0926 03:45:19.190595 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 03:45:19.213153 (+ 22558us) server_negotiation.cc:407] Connection header received
0926 03:45:19.213441 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:45:19.213446 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:45:19.213555 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:45:19.213641 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:45:19.215437 (+ 1796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:45:19.216728 (+ 1291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:45:19.218603 (+ 1875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:45:19.219425 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:45:19.219535 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:45:19.224949 (+ 5414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:45:19.224961 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:45:19.224979 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:45:19.225024 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:45:19.229053 (+ 4029us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:45:19.230012 (+ 959us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:45:19.230023 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:45:19.230027 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:45:19.230216 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:45:19.230681 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:45:19.230687 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:45:19.230690 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:45:19.231173 (+ 483us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:45:19.231436 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0926 03:45:19.232034 (+ 598us) server_negotiation.cc:299] Negotiation successful
0926 03:45:19.232304 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 03:45:19.234309 11300 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:45:19.234445 11103 catalog_manager.cc:4841] TS 0adabfbeaccc4d208b18dd145d3805e7 (127.0.194.194:46025): tablet ab90fb7f22ae4bcbb3e54e6b67cc57c4 (table test-table [id=1e3be51ce7f44be3b65170f8d2d8f487]) successfully deleted
I20260926 03:45:19.330884 779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu with pid 11177
I20260926 03:45:19.349215 779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu pid 11311
Sep 26 03:45:19 dist-test-slave-n2th krb5kdc[11071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394316, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.194.194@KRBTEST.COM
I20260926 03:45:19.364073 11589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:45:19.350047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55482 (local address 127.0.194.194:46025)
0926 03:45:19.350252 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0926 03:45:19.350260 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:45:19.350287 (+ 27us) server_negotiation.cc:407] Connection header received
0926 03:45:19.350519 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:45:19.350523 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:45:19.350585 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:45:19.350689 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:45:19.351972 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:45:19.352763 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:45:19.354021 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:45:19.354423 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:45:19.354559 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:45:19.358663 (+ 4104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:45:19.358679 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:45:19.358696 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:45:19.358804 (+ 108us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:45:19.361666 (+ 2862us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:45:19.362309 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:45:19.362316 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:45:19.362319 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:45:19.362440 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:45:19.362809 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:45:19.362812 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:45:19.362814 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:45:19.363163 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:45:19.363379 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0926 03:45:19.363661 (+ 282us) server_negotiation.cc:299] Negotiation successful
0926 03:45:19.363815 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 03:45:19.364827 11434 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:45:19.453357 779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu with pid 11311
I20260926 03:45:19.471449 779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu pid 11445
Sep 26 03:45:19 dist-test-slave-n2th krb5kdc[11071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394316, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.194.195@KRBTEST.COM
I20260926 03:45:19.486928 11588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:45:19.472228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34434 (local address 127.0.194.195:44009)
0926 03:45:19.472416 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0926 03:45:19.472424 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:45:19.472522 (+ 98us) server_negotiation.cc:407] Connection header received
0926 03:45:19.472787 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:45:19.472791 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:45:19.472864 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:45:19.472972 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:45:19.474226 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:45:19.474999 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:45:19.476258 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:45:19.476646 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:45:19.476782 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:45:19.481200 (+ 4418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:45:19.481215 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:45:19.481232 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:45:19.481270 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:45:19.483978 (+ 2708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:45:19.484694 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:45:19.484702 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:45:19.484704 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:45:19.484848 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:45:19.485228 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:45:19.485235 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:45:19.485238 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:45:19.485606 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:45:19.485896 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0926 03:45:19.486217 (+ 321us) server_negotiation.cc:299] Negotiation successful
0926 03:45:19.486437 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 03:45:19.487586 11568 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:45:19.576930 779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu with pid 11445
I20260926 03:45:19.594157 779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu pid 11084
I20260926 03:45:19.595314 11146 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:45:19.704042 779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu with pid 11084
2026-09-26T03:45:19Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4347 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790394221564982-779-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:45:19 dist-test-slave-n2th krb5kdc[11625](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:45:19 dist-test-slave-n2th krb5kdc[11625](info): set up 2 sockets
Sep 26 03:45:19 dist-test-slave-n2th krb5kdc[11625](info): commencing operation
krb5kdc: starting...
W20260926 03:45:20.807760 779 mini_kdc.cc:121] Time spent starting KDC: real 1.058s user 0.004s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790394221564982-779-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790394221564982-779-0/krb5kdc/test-user.keytab.
Sep 26 03:45:20 dist-test-slave-n2th krb5kdc[11625](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394320, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:45:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:45:20Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.194.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.194.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.194.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790394221564982-779-0/krb5kdc/rangeradmin_127.0.194.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.194.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790394221564982-779-0/krb5kdc/rangeradmin_127.0.194.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.194.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.194.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.194.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790394221564982-779-0/krb5kdc/rangerlookup_127.0.194.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.194.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790394221564982-779-0/krb5kdc/rangerlookup_127.0.194.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.194.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.194.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.194.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790394221564982-779-0/krb5kdc/HTTP_127.0.194.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.194.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790394221564982-779-0/krb5kdc/HTTP_127.0.194.212@KRBTEST.COM.keytab.
I20260926 03:45:21.146713 779 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790394221564982-779-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-tasktGRWtf/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790394221564982-779-0/minicluster-data/postgres -l logfile start
2026-09-26 03:45:23.634 UTC [11655] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-26 03:45:23.634 UTC [11655] LOG: listening on IPv4 address "127.0.194.212", port 37405
2026-09-26 03:45:23.637 UTC [11655] LOG: listening on Unix socket "/tmp/.s.PGSQL.37405"
2026-09-26 03:45:23.663 UTC [11658] LOG: database system was shut down at 2026-09-26 03:45:22 UTC
2026-09-26 03:45:23.666 UTC [11655] LOG: database system is ready to accept connections
I20260926 03:45:24.647521 779 mini_postgres.cc:96] Postgres bound to 37405
2026-09-26 03:45:24.667 UTC [11666] FATAL: database "slave" does not exist
127.0.194.212:37405 - accepting connections
I20260926 03:45:24.668386 779 mini_ranger.cc:161] Starting Ranger...
I20260926 03:45:24.695082 779 mini_ranger.cc:84] Created miniranger Postgres user
I20260926 03:45:25.163928 779 mini_ranger.cc:87] Created ranger Postgres database
I20260926 03:45:25.164040 779 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790394221564982-779-0/minicluster-data/ranger-admin
2026-09-26 03:45:25,462 [I] DB FLAVOR :POSTGRES
2026-09-26 03:45:25,462 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 03:45:25,462 [I] Checking connection..
2026-09-26 03:45:25,462 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktGRWtf/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktGRWtf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.194.212:37405/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-26 03:45:25,733 [I] Checking connection passed.
2026-09-26 03:45:25,733 [I] --------- Verifying version history table ---------
2026-09-26 03:45:25,733 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktGRWtf/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktGRWtf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.194.212:37405/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 03:45:26,001 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 03:45:26,002 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktGRWtf/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktGRWtf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.194.212:37405/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 03:45:26,252 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 03:45:26,253 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-26 03:45:26,253 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktGRWtf/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktGRWtf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.194.212:37405/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasktGRWtf/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-26 03:45:26.512 UTC [11730] WARNING: there is no transaction in progress
2026-09-26 03:45:26,519 [I] create_dbversion_catalog.sql file imported successfully
2026-09-26 03:45:26,519 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktGRWtf/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktGRWtf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.194.212:37405/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 03:45:26,781 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-26 03:45:26,781 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 03:45:26,782 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktGRWtf/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktGRWtf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.194.212:37405/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-26 03:45:27,036 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktGRWtf/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktGRWtf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.194.212:37405/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-26 03:45:27,302 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktGRWtf/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktGRWtf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.194.212:37405/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-n2th.c.gcp-upstream.internal','N') ;"
2026-09-26 03:45:27,538 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktGRWtf/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktGRWtf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.194.212:37405/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 03:45:27,804 [I] Table x_portal_user does not exist in database ranger
2026-09-26 03:45:27,804 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktGRWtf/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktGRWtf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.194.212:37405/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 03:45:28,071 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-26 03:45:28,071 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktGRWtf/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktGRWtf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.194.212:37405/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 03:45:28,340 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktGRWtf/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktGRWtf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.194.212:37405/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 03:45:28,584 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-26 03:45:28,585 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktGRWtf/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktGRWtf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.194.212:37405/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasktGRWtf/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-26 03:45:29.566 UTC [11838] WARNING: there is no transaction in progress
2026-09-26 03:45:29.583 UTC [11838] WARNING: there is no transaction in progress
2026-09-26 03:45:29.600 UTC [11838] WARNING: there is no transaction in progress
2026-09-26 03:45:29.616 UTC [11838] WARNING: there is no transaction in progress
2026-09-26 03:45:29.633 UTC [11838] WARNING: there is no transaction in progress
2026-09-26 03:45:29.652 UTC [11838] WARNING: there is no transaction in progress
2026-09-26 03:45:29.709 UTC [11838] WARNING: there is no transaction in progress
2026-09-26 03:45:29.716 UTC [11838] WARNING: there is no transaction in progress
2026-09-26 03:45:29.731 UTC [11838] WARNING: there is no transaction in progress
2026-09-26 03:45:29.741 UTC [11838] WARNING: there is no transaction in progress
2026-09-26 03:45:29.752 UTC [11838] WARNING: there is no transaction in progress
2026-09-26 03:45:29.765 UTC [11838] WARNING: there is no transaction in progress
2026-09-26 03:45:29.775 UTC [11838] WARNING: there is no transaction in progress
2026-09-26 03:45:29.782 UTC [11838] WARNING: there is no transaction in progress
2026-09-26 03:45:29.789 UTC [11838] WARNING: there is no transaction in progress
2026-09-26 03:45:29.795 UTC [11838] WARNING: there is no transaction in progress
2026-09-26 03:45:30,469 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-26 03:45:30,469 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktGRWtf/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktGRWtf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.194.212:37405/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-26 03:45:30,717 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-26 03:45:30,718 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktGRWtf/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktGRWtf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.194.212:37405/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 03:45:30,957 [I] Table x_portal_user already exists in database 'ranger'
2026-09-26 03:45:30,957 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktGRWtf/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktGRWtf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.194.212:37405/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 03:45:31,199 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-26 03:45:31,199 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktGRWtf/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktGRWtf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.194.212:37405/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 03:45:31,454 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktGRWtf/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktGRWtf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.194.212:37405/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 03:45:31,700 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktGRWtf/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktGRWtf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.194.212:37405/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-n2th.c.gcp-upstream.internal';"
2026-09-26 03:45:31,945 [I] CORE_DB_SCHEMA import status has been updated
2026-09-26 03:45:31,945 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktGRWtf/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktGRWtf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.194.212:37405/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-26 03:45:32,179 [I] DB_PATCHES have already been applied
I20260926 03:45:32.185376 779 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790394221564982-779-0/minicluster-data/ranger-admin:/tmp/dist-test-tasktGRWtf/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-tasktGRWtf/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-tasktGRWtf/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktGRWtf/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260926 03:45:32.185503 779 mini_ranger.cc:193] Using host: 127.0.194.212
I20260926 03:45:32.201128 779 mini_ranger.cc:227] Ranger admin URL: http://127.0.194.212:51385
Sep 26, 2026 3:45:32 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 3:45:32 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 3:45:32 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790394221564982-779-0/minicluster-data/ranger-admin/ews/webapp
Sep 26, 2026 3:45:32 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790394221564982-779-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 26, 2026 3:45:32 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790394221564982-779-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 26, 2026 3:45:33 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790394221564982-779-0/minicluster-data/ranger-admin/ews/webapp].
Sep 26, 2026 3:45:33 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260926 03:45:33.353381 779 mini_ranger.cc:160] Time spent starting Ranger: real 8.685s user 0.000s sys 0.042s
Sep 26, 2026 3:45:53 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 26, 2026 3:45:53 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.MetricsREST
Sep 26, 2026 3:45:53 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 26, 2026 3:45:53 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 26, 2026 3:45:53 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 26, 2026 3:45:53 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 26, 2026 3:45:53 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 26, 2026 3:45:53 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 26, 2026 3:45:53 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 26, 2026 3:45:53 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 26, 2026 3:45:53 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 26, 2026 3:45:53 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 26, 2026 3:45:53 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 26, 2026 3:45:53 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 26, 2026 3:45:53 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 26, 2026 3:45:53 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 26, 2026 3:45:53 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 26, 2026 3:45:54 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260926 03:45:55.225931 779 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.194.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.194.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.194.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790394221564982-779-0/krb5kdc/rangerkms_127.0.194.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.194.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790394221564982-779-0/krb5kdc/rangerkms_127.0.194.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.194.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.194.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.194.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790394221564982-779-0/krb5kdc/HTTP_127.0.194.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.194.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790394221564982-779-0/krb5kdc/HTTP_127.0.194.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 26 03:45:55 dist-test-slave-n2th krb5kdc[11625](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394355, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260926 03:45:55.417950 779 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260926 03:45:55.442934 779 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260926 03:45:56.264658 779 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260926 03:45:56.264801 779 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790394221564982-779-0/minicluster-data/ranger-kms
I20260926 03:45:56.264928 779 mini_ranger_kms.cc:227] Using postgres at 127.0.194.212:37405
2026-09-26 03:45:56,696 [I] DB FLAVOR :POSTGRES
2026-09-26 03:45:56,696 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 03:45:56,696 [I] Checking connection
2026-09-26 03:45:56,696 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktGRWtf/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790394221564982-779-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.194.212:37405/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-26 03:45:56,940 [I] connection success
2026-09-26 03:45:56,940 [I] --------- Verifying Ranger DB tables ---------
2026-09-26 03:45:56,940 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-26 03:45:56,941 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktGRWtf/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790394221564982-779-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.194.212:37405/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-26 03:45:57,191 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-26 03:45:57,191 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 03:45:57,191 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-26 03:45:57,191 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktGRWtf/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790394221564982-779-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.194.212:37405/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790394221564982-779-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-26 03:45:57,450 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260926 03:45:57.909181 779 mini_ranger_kms.cc:310] Created kms service
I20260926 03:45:58.086668 779 mini_ranger_kms.cc:326] Created kudu user
I20260926 03:45:58.132890 779 mini_ranger_kms.cc:343] Created rangerkms user
Sep 26 03:45:58 dist-test-slave-n2th krb5kdc[11625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394355, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.194.212@KRBTEST.COM
I20260926 03:45:58.554425 779 mini_ranger_kms.cc:384] Added ranger policy
I20260926 03:45:58.554808 779 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790394221564982-779-0/minicluster-data/ranger-kms:/tmp/dist-test-tasktGRWtf/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktGRWtf/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-tasktGRWtf/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260926 03:45:58.554867 779 mini_ranger_kms.cc:240] Using host: 127.0.194.212
I20260926 03:45:58.576565 779 mini_ranger_kms.cc:276] Ranger KMS PID: 12018
I20260926 03:45:58.576771 779 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.194.212:48877
Sep 26, 2026 3:45:59 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 3:45:59 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 3:45:59 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790394221564982-779-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 26, 2026 3:45:59 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790394221564982-779-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 26, 2026 3:46:00 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790394221564982-779-0/minicluster-data/ranger-kms/ews/webapp].
Sep 26, 2026 3:46:00 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260926 03:46:00.989181 779 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.571s user 0.120s sys 0.340s
I20260926 03:46:00.989444 779 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260926 03:46:00.989663 779 mini_ranger_kms.cc:401] 127.0.194.212:48877/kms/v1/keys
Sep 26 03:46:01 dist-test-slave-n2th krb5kdc[11625](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790394361, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.194.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 26 03:46:01 dist-test-slave-n2th krb5kdc[11625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394361, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.194.212@KRBTEST.COM for HTTP/127.0.194.212@KRBTEST.COM
Sep 26 03:46:01 dist-test-slave-n2th krb5kdc[11625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394361, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.194.212@KRBTEST.COM for HTTP/127.0.194.212@KRBTEST.COM
Sep 26 03:46:04 dist-test-slave-n2th krb5kdc[11625](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394364, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.194.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.194.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.194.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790394221564982-779-0/krb5kdc/kudu_127.0.194.254.keytab.
Entry for principal kudu/127.0.194.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790394221564982-779-0/krb5kdc/kudu_127.0.194.254.keytab.
I20260926 03:46:04.218078 779 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
/tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790394221564982-779-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790394221564982-779-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790394221564982-779-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790394221564982-779-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.194.254:42187
--webserver_interface=127.0.194.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790394221564982-779-0/krb5kdc/kudu_127.0.194.254.keytab
--principal=kudu/127.0.194.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.194.212:37757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.194.254:42187
--ranger_config_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790394221564982-779-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.194.212:48877/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790394221564982-779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790394221564982-779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790394221564982-779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:46:04.567116 12069 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:46:04.567828 12069 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:46:04.567987 12069 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:46:04.575421 12069 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:46:04.575541 12069 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:46:04.575629 12069 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:46:04.575717 12069 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:46:04.575805 12069 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:46:04.586392 12069 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.194.212:37757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790394221564982-779-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790394221564982-779-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.194.212:48877/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.194.254:42187
--ranger_config_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790394221564982-779-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790394221564982-779-0/krb5kdc/kudu_127.0.194.254.keytab
--principal=kudu/127.0.194.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.194.254:42187
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790394221564982-779-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.194.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790394221564982-779-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:46:04.587797 12069 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:46:04.589195 12069 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:46:04.603091 12074 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:46:04.603072 12075 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:46:04.603030 12077 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:46:04.603809 12069 server_base.cc:1034] running on GCE node
I20260926 03:46:04.604743 12069 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:46:04.606379 12069 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:46:04.607617 12069 hybrid_clock.cc:648] HybridClock initialized: now 1790394364607553 us; error 74 us; skew 500 ppm
I20260926 03:46:04.608009 12069 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:46:04 dist-test-slave-n2th krb5kdc[11625](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394364, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:46:04.619385 12069 init.cc:377] Logged in from keytab as kudu/127.0.194.254@KRBTEST.COM (short username kudu)
I20260926 03:46:04.621214 12069 webserver.cc:460] Webserver started at http://127.0.194.254:33559/ using document root <none> and password file <none>
I20260926 03:46:04.621977 12069 fs_manager.cc:359] Metadata directory not provided
I20260926 03:46:04.622129 12069 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:46:04.622468 12069 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 03:46:04 dist-test-slave-n2th krb5kdc[11625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394364, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.254@KRBTEST.COM for HTTP/127.0.194.212@KRBTEST.COM
I20260926 03:46:04.677129 12069 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790394221564982-779-0/minicluster-data/master-0/data/instance:
uuid: "b047ae3c851b4fbd8262e06259a8c9b0"
format_stamp: "Formatted at 2026-09-26 03:46:04 on dist-test-slave-n2th"
server_key: "275beb14d1b3937f963a71fa79c661f9"
server_key_iv: "210203a341a6827ca6863e38c8985c0f"
server_key_version: "kuduclusterkey@0"
I20260926 03:46:04.678026 12069 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790394221564982-779-0/minicluster-data/master-0/wal/instance:
uuid: "b047ae3c851b4fbd8262e06259a8c9b0"
format_stamp: "Formatted at 2026-09-26 03:46:04 on dist-test-slave-n2th"
server_key: "275beb14d1b3937f963a71fa79c661f9"
server_key_iv: "210203a341a6827ca6863e38c8985c0f"
server_key_version: "kuduclusterkey@0"
I20260926 03:46:04.683923 12069 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260926 03:46:04.704368 12084 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:46:04.706027 12069 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260926 03:46:04.706235 12069 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790394221564982-779-0/minicluster-data/master-0/data,/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790394221564982-779-0/minicluster-data/master-0/wal
uuid: "b047ae3c851b4fbd8262e06259a8c9b0"
format_stamp: "Formatted at 2026-09-26 03:46:04 on dist-test-slave-n2th"
server_key: "275beb14d1b3937f963a71fa79c661f9"
server_key_iv: "210203a341a6827ca6863e38c8985c0f"
server_key_version: "kuduclusterkey@0"
I20260926 03:46:04.706480 12069 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790394221564982-779-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790394221564982-779-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790394221564982-779-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:46:04.721761 12069 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:46:04.725872 12069 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:46:04.726188 12069 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:46:04.746140 12069 rpc_server.cc:307] RPC server started. Bound to: 127.0.194.254:42187
I20260926 03:46:04.746163 12135 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.194.254:42187 every 8 connection(s)
I20260926 03:46:04.747576 12069 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790394221564982-779-0/minicluster-data/master-0/data/info.pb
I20260926 03:46:04.751531 12136 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:46:04.756364 779 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu as pid 12069
I20260926 03:46:04.756582 779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790394221564982-779-0/minicluster-data/master-0/wal/instance
I20260926 03:46:04.757265 779 external_mini_cluster.cc:1428] Setting key 0d71c13efb99b955bc105bd053ec4bd3
I20260926 03:46:04.764716 12136 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b047ae3c851b4fbd8262e06259a8c9b0: Bootstrap starting.
I20260926 03:46:04.768464 12136 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b047ae3c851b4fbd8262e06259a8c9b0: Neither blocks nor log segments found. Creating new log.
I20260926 03:46:04.769850 12136 log.cc:824] T 00000000000000000000000000000000 P b047ae3c851b4fbd8262e06259a8c9b0: Log is configured to *not* fsync() on all Append() calls
I20260926 03:46:04.772884 12136 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b047ae3c851b4fbd8262e06259a8c9b0: No bootstrap required, opened a new log
Sep 26 03:46:04 dist-test-slave-n2th krb5kdc[11625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394364, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.194.254@KRBTEST.COM
I20260926 03:46:04.782980 12139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:46:04.759502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55216 (local address 127.0.194.254:42187)
0926 03:46:04.760387 (+ 885us) server_negotiation.cc:206] Beginning negotiation
0926 03:46:04.760405 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0926 03:46:04.760460 (+ 55us) server_negotiation.cc:407] Connection header received
0926 03:46:04.763010 (+ 2550us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:46:04.763041 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:46:04.763592 (+ 551us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:46:04.764049 (+ 457us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:46:04.765315 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:46:04.767382 (+ 2067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:46:04.768691 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:46:04.769749 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:46:04.769944 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:46:04.774002 (+ 4058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:46:04.774016 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:46:04.774041 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:46:04.774089 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:46:04.778049 (+ 3960us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:46:04.778897 (+ 848us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:46:04.778909 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:46:04.778920 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:46:04.779131 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:46:04.779529 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:46:04.779535 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:46:04.779540 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:46:04.780327 (+ 787us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:46:04.780648 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0926 03:46:04.781008 (+ 360us) server_negotiation.cc:299] Negotiation successful
0926 03:46:04.781482 (+ 474us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":626,"thread_start_us":402,"threads_started":1}
I20260926 03:46:04.784211 12136 raft_consensus.cc:348] T 00000000000000000000000000000000 P b047ae3c851b4fbd8262e06259a8c9b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b047ae3c851b4fbd8262e06259a8c9b0" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 42187 } }
I20260926 03:46:04.784582 12136 raft_consensus.cc:374] T 00000000000000000000000000000000 P b047ae3c851b4fbd8262e06259a8c9b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:46:04.784739 12136 raft_consensus.cc:729] T 00000000000000000000000000000000 P b047ae3c851b4fbd8262e06259a8c9b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b047ae3c851b4fbd8262e06259a8c9b0, State: Initialized, Role: FOLLOWER
I20260926 03:46:04.785333 12136 consensus_queue.cc:260] T 00000000000000000000000000000000 P b047ae3c851b4fbd8262e06259a8c9b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b047ae3c851b4fbd8262e06259a8c9b0" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 42187 } }
I20260926 03:46:04.785571 12136 raft_consensus.cc:388] T 00000000000000000000000000000000 P b047ae3c851b4fbd8262e06259a8c9b0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:46:04.785715 12136 raft_consensus.cc:482] T 00000000000000000000000000000000 P b047ae3c851b4fbd8262e06259a8c9b0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:46:04.785894 12136 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b047ae3c851b4fbd8262e06259a8c9b0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:46:04.788586 12136 raft_consensus.cc:504] T 00000000000000000000000000000000 P b047ae3c851b4fbd8262e06259a8c9b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b047ae3c851b4fbd8262e06259a8c9b0" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 42187 } }
I20260926 03:46:04.789366 12136 leader_election.cc:302] T 00000000000000000000000000000000 P b047ae3c851b4fbd8262e06259a8c9b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b047ae3c851b4fbd8262e06259a8c9b0; no voters:
I20260926 03:46:04.790086 12136 leader_election.cc:288] T 00000000000000000000000000000000 P b047ae3c851b4fbd8262e06259a8c9b0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:46:04.790295 12141 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b047ae3c851b4fbd8262e06259a8c9b0 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:46:04.791433 12141 raft_consensus.cc:686] T 00000000000000000000000000000000 P b047ae3c851b4fbd8262e06259a8c9b0 [term 1 LEADER]: Becoming Leader. State: Replica: b047ae3c851b4fbd8262e06259a8c9b0, State: Running, Role: LEADER
I20260926 03:46:04.792050 12141 consensus_queue.cc:237] T 00000000000000000000000000000000 P b047ae3c851b4fbd8262e06259a8c9b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b047ae3c851b4fbd8262e06259a8c9b0" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 42187 } }
I20260926 03:46:04.792837 12136 sys_catalog.cc:564] T 00000000000000000000000000000000 P b047ae3c851b4fbd8262e06259a8c9b0 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:46:04.795445 12143 sys_catalog.cc:455] T 00000000000000000000000000000000 P b047ae3c851b4fbd8262e06259a8c9b0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b047ae3c851b4fbd8262e06259a8c9b0. Latest consensus state: current_term: 1 leader_uuid: "b047ae3c851b4fbd8262e06259a8c9b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b047ae3c851b4fbd8262e06259a8c9b0" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 42187 } } }
I20260926 03:46:04.795917 12143 sys_catalog.cc:458] T 00000000000000000000000000000000 P b047ae3c851b4fbd8262e06259a8c9b0 [sys.catalog]: This master's current role is: LEADER
I20260926 03:46:04.796510 12142 sys_catalog.cc:455] T 00000000000000000000000000000000 P b047ae3c851b4fbd8262e06259a8c9b0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b047ae3c851b4fbd8262e06259a8c9b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b047ae3c851b4fbd8262e06259a8c9b0" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 42187 } } }
I20260926 03:46:04.796903 12142 sys_catalog.cc:458] T 00000000000000000000000000000000 P b047ae3c851b4fbd8262e06259a8c9b0 [sys.catalog]: This master's current role is: LEADER
I20260926 03:46:04.799113 12150 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:46:04.805697 12150 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:46:04.806952 12136 ranger_client.cc:318] Using new properties file: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790394221564982-779-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260926 03:46:04.827833 12150 catalog_manager.cc:1347] Generated new cluster ID: cd32e4775e9442ec9e4869aebb368d13
I20260926 03:46:04.827986 12150 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:46:04.857254 12150 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:46:04.859419 12150 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:46:04.871657 12150 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b047ae3c851b4fbd8262e06259a8c9b0: Generated new TSK 0
I20260926 03:46:04.872859 12150 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.194.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.194.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.194.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790394221564982-779-0/krb5kdc/kudu_127.0.194.193.keytab.
Entry for principal kudu/127.0.194.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790394221564982-779-0/krb5kdc/kudu_127.0.194.193.keytab.
I20260926 03:46:05.598971 779 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
/tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790394221564982-779-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790394221564982-779-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790394221564982-779-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790394221564982-779-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.194.212:48877/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.194.193:0
--local_ip_for_outbound_sockets=127.0.194.193
--webserver_interface=127.0.194.193
--webserver_port=0
--tserver_master_addrs=127.0.194.254:42187
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790394221564982-779-0/krb5kdc/kudu_127.0.194.193.keytab
--principal=kudu/127.0.194.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.194.212:37757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790394221564982-779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790394221564982-779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790394221564982-779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 03:46:05 dist-test-slave-n2th krb5kdc[11625](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790394365, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260926 03:46:06.092617 12181 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:46:06.093070 12181 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:46:06.093302 12181 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:46:06.105643 12181 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:46:06.105808 12181 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:46:06.106067 12181 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.194.193
I20260926 03:46:06.122897 12181 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.194.212:37757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790394221564982-779-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790394221564982-779-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.194.212:48877/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790394221564982-779-0/krb5kdc/kudu_127.0.194.193.keytab
--principal=kudu/127.0.194.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.194.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790394221564982-779-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.194.193
--webserver_port=0
--tserver_master_addrs=127.0.194.254:42187
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.194.193
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790394221564982-779-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:46:06.124792 12181 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:46:06.126684 12181 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:46:06.147607 12191 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:46:06.147607 12192 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:46:06.163671 12194 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:46:06.164777 12181 server_base.cc:1034] running on GCE node
I20260926 03:46:06.165602 12181 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:46:06.166868 12181 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:46:06.168115 12181 hybrid_clock.cc:648] HybridClock initialized: now 1790394366168031 us; error 79 us; skew 500 ppm
I20260926 03:46:06.168555 12181 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:46:06 dist-test-slave-n2th krb5kdc[11625](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394366, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:46:06.172072 12181 init.cc:377] Logged in from keytab as kudu/127.0.194.193@KRBTEST.COM (short username kudu)
I20260926 03:46:06.176168 12181 webserver.cc:460] Webserver started at http://127.0.194.193:40867/ using document root <none> and password file <none>
I20260926 03:46:06.177017 12181 fs_manager.cc:359] Metadata directory not provided
I20260926 03:46:06.177174 12181 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:46:06.177525 12181 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 03:46:06 dist-test-slave-n2th krb5kdc[11625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394366, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.193@KRBTEST.COM for HTTP/127.0.194.212@KRBTEST.COM
I20260926 03:46:06.197702 12181 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790394221564982-779-0/minicluster-data/ts-0/data/instance:
uuid: "d76665909ce842df9635c0f625f7c795"
format_stamp: "Formatted at 2026-09-26 03:46:06 on dist-test-slave-n2th"
server_key: "de6f6a12ab6de96cef334a14777caab1"
server_key_iv: "812c47ae2e48d80f8c2cfba03ece782f"
server_key_version: "kuduclusterkey@0"
I20260926 03:46:06.199241 12181 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790394221564982-779-0/minicluster-data/ts-0/wal/instance:
uuid: "d76665909ce842df9635c0f625f7c795"
format_stamp: "Formatted at 2026-09-26 03:46:06 on dist-test-slave-n2th"
server_key: "de6f6a12ab6de96cef334a14777caab1"
server_key_iv: "812c47ae2e48d80f8c2cfba03ece782f"
server_key_version: "kuduclusterkey@0"
I20260926 03:46:06.207901 12181 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.005s sys 0.001s
I20260926 03:46:06.229867 12201 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:46:06.232023 12181 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260926 03:46:06.232273 12181 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790394221564982-779-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790394221564982-779-0/minicluster-data/ts-0/wal
uuid: "d76665909ce842df9635c0f625f7c795"
format_stamp: "Formatted at 2026-09-26 03:46:06 on dist-test-slave-n2th"
server_key: "de6f6a12ab6de96cef334a14777caab1"
server_key_iv: "812c47ae2e48d80f8c2cfba03ece782f"
server_key_version: "kuduclusterkey@0"
I20260926 03:46:06.232579 12181 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790394221564982-779-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790394221564982-779-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790394221564982-779-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:46:06.268990 12181 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:46:06.274557 12181 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:46:06.274984 12181 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:46:06.276257 12181 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:46:06.279655 12181 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:46:06.279767 12181 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:46:06.279906 12181 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:46:06.279973 12181 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:46:06.359113 12181 rpc_server.cc:307] RPC server started. Bound to: 127.0.194.193:32843
I20260926 03:46:06.359391 12313 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.194.193:32843 every 8 connection(s)
I20260926 03:46:06.360945 12181 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790394221564982-779-0/minicluster-data/ts-0/data/info.pb
I20260926 03:46:06.363225 779 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu as pid 12181
I20260926 03:46:06.363451 779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790394221564982-779-0/minicluster-data/ts-0/wal/instance
I20260926 03:46:06.364326 779 external_mini_cluster.cc:1428] Setting key f44540388147c346c519603e5d56809b
Sep 26 03:46:06 dist-test-slave-n2th krb5kdc[11625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394366, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.193@KRBTEST.COM for kudu/127.0.194.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.194.194@KRBTEST.COM; defaulting to no policy
I20260926 03:46:06.411247 12316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:46:06.371202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.193:57833 (local address 127.0.194.254:42187)
0926 03:46:06.383752 (+ 12550us) server_negotiation.cc:206] Beginning negotiation
0926 03:46:06.383758 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:46:06.383781 (+ 23us) server_negotiation.cc:407] Connection header received
0926 03:46:06.383846 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:46:06.383852 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:46:06.383939 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:46:06.384068 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:46:06.386594 (+ 2526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:46:06.388222 (+ 1628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:46:06.392251 (+ 4029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:46:06.392964 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:46:06.393536 (+ 572us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:46:06.402371 (+ 8835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:46:06.402380 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:46:06.402385 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:46:06.402449 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:46:06.405861 (+ 3412us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:46:06.407831 (+ 1970us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:46:06.407838 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:46:06.407841 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:46:06.408047 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:46:06.408726 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:46:06.408732 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:46:06.408736 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:46:06.409166 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:46:06.409306 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0926 03:46:06.410504 (+ 1198us) server_negotiation.cc:299] Negotiation successful
0926 03:46:06.410829 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12411,"thread_start_us":12981,"threads_started":1}
I20260926 03:46:06.413463 12315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:46:06.370565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:42187 (local address 127.0.194.193:57833)
0926 03:46:06.377410 (+ 6845us) negotiation.cc:107] Waiting for socket to connect
0926 03:46:06.377473 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0926 03:46:06.382136 (+ 4663us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:46:06.384653 (+ 2517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:46:06.384668 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:46:06.385370 (+ 702us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:46:06.386320 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:46:06.386348 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:46:06.389033 (+ 2685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:46:06.389039 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:46:06.392005 (+ 2966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:46:06.392026 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:46:06.393887 (+ 1861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:46:06.393954 (+ 67us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:46:06.394293 (+ 339us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:46:06.396050 (+ 1757us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:46:06.396083 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:46:06.401936 (+ 5853us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:46:06.406097 (+ 4161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:46:06.406108 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:46:06.406129 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:46:06.406807 (+ 678us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:46:06.408237 (+ 1430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:46:06.408243 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:46:06.408247 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:46:06.408573 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:46:06.409428 (+ 855us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:46:06.409438 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:46:06.409977 (+ 539us) client_negotiation.cc:769] Sending connection context
0926 03:46:06.410978 (+ 1001us) client_negotiation.cc:240] Negotiation successful
0926 03:46:06.411866 (+ 888us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6468,"thread_start_us":6192,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.194.194@KRBTEST.COM" created.
I20260926 03:46:06.415731 12314 heartbeater.cc:347] Connected to a master server at 127.0.194.254:42187
I20260926 03:46:06.416061 12314 heartbeater.cc:464] Registering TS with master...
I20260926 03:46:06.416903 12314 heartbeater.cc:511] Master 127.0.194.254:42187 requested a full tablet report, sending...
I20260926 03:46:06.419406 12101 ts_manager.cc:194] Registered new tserver with Master: d76665909ce842df9635c0f625f7c795 (127.0.194.193:32843)
I20260926 03:46:06.421510 12101 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.194.193@KRBTEST.COM'} at 127.0.194.193:57833
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.194.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790394221564982-779-0/krb5kdc/kudu_127.0.194.194.keytab.
Entry for principal kudu/127.0.194.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790394221564982-779-0/krb5kdc/kudu_127.0.194.194.keytab.
I20260926 03:46:06.451830 779 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
/tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790394221564982-779-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790394221564982-779-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790394221564982-779-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790394221564982-779-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.194.212:48877/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.194.194:0
--local_ip_for_outbound_sockets=127.0.194.194
--webserver_interface=127.0.194.194
--webserver_port=0
--tserver_master_addrs=127.0.194.254:42187
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790394221564982-779-0/krb5kdc/kudu_127.0.194.194.keytab
--principal=kudu/127.0.194.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.194.212:37757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790394221564982-779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790394221564982-779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790394221564982-779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 03:46:06 dist-test-slave-n2th krb5kdc[11625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394365, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.254@KRBTEST.COM for HTTP/127.0.194.212@KRBTEST.COM
Sep 26 03:46:06 dist-test-slave-n2th krb5kdc[11625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394365, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.254@KRBTEST.COM for HTTP/127.0.194.212@KRBTEST.COM
W20260926 03:46:06.956552 12320 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:46:06.957090 12320 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:46:06.957294 12320 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:46:06.968991 12320 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:46:06.969242 12320 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:46:06.969477 12320 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.194.194
I20260926 03:46:06.985814 12320 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.194.212:37757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790394221564982-779-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790394221564982-779-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.194.212:48877/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790394221564982-779-0/krb5kdc/kudu_127.0.194.194.keytab
--principal=kudu/127.0.194.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.194.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790394221564982-779-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.194.194
--webserver_port=0
--tserver_master_addrs=127.0.194.254:42187
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.194.194
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790394221564982-779-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:46:06.987911 12320 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:46:06.989773 12320 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:46:07.031339 12328 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:46:07.034502 12327 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:46:07.035053 12331 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:46:07.038261 12320 server_base.cc:1034] running on GCE node
I20260926 03:46:07.039383 12320 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:46:07.048259 12320 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:46:07.049762 12320 hybrid_clock.cc:648] HybridClock initialized: now 1790394367049637 us; error 139 us; skew 500 ppm
I20260926 03:46:07.050307 12320 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:46:07 dist-test-slave-n2th krb5kdc[11625](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394367, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:46:07.056169 12320 init.cc:377] Logged in from keytab as kudu/127.0.194.194@KRBTEST.COM (short username kudu)
I20260926 03:46:07.092562 12320 webserver.cc:460] Webserver started at http://127.0.194.194:42919/ using document root <none> and password file <none>
I20260926 03:46:07.093743 12320 fs_manager.cc:359] Metadata directory not provided
I20260926 03:46:07.093978 12320 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:46:07.094426 12320 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 03:46:07 dist-test-slave-n2th krb5kdc[11625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394367, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.194@KRBTEST.COM for HTTP/127.0.194.212@KRBTEST.COM
I20260926 03:46:07.136518 12320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790394221564982-779-0/minicluster-data/ts-1/data/instance:
uuid: "c17a512c73c942db9c14966832291317"
format_stamp: "Formatted at 2026-09-26 03:46:07 on dist-test-slave-n2th"
server_key: "310e80833b6fca022fb6c010073a3f69"
server_key_iv: "605109f1c66dafcd95f5a8e4be8e5fa0"
server_key_version: "kuduclusterkey@0"
I20260926 03:46:07.137648 12320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790394221564982-779-0/minicluster-data/ts-1/wal/instance:
uuid: "c17a512c73c942db9c14966832291317"
format_stamp: "Formatted at 2026-09-26 03:46:07 on dist-test-slave-n2th"
server_key: "310e80833b6fca022fb6c010073a3f69"
server_key_iv: "605109f1c66dafcd95f5a8e4be8e5fa0"
server_key_version: "kuduclusterkey@0"
I20260926 03:46:07.144867 12320 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.000s
I20260926 03:46:07.163846 12344 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:46:07.165805 12320 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260926 03:46:07.166077 12320 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790394221564982-779-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790394221564982-779-0/minicluster-data/ts-1/wal
uuid: "c17a512c73c942db9c14966832291317"
format_stamp: "Formatted at 2026-09-26 03:46:07 on dist-test-slave-n2th"
server_key: "310e80833b6fca022fb6c010073a3f69"
server_key_iv: "605109f1c66dafcd95f5a8e4be8e5fa0"
server_key_version: "kuduclusterkey@0"
I20260926 03:46:07.166416 12320 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790394221564982-779-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790394221564982-779-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790394221564982-779-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:46:07.200382 12320 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:46:07.205976 12320 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:46:07.206311 12320 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:46:07.207484 12320 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:46:07.208905 12320 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:46:07.209009 12320 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:46:07.209160 12320 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:46:07.209232 12320 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:46:07.247570 12320 rpc_server.cc:307] RPC server started. Bound to: 127.0.194.194:37315
I20260926 03:46:07.247588 12456 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.194.194:37315 every 8 connection(s)
I20260926 03:46:07.248950 12320 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790394221564982-779-0/minicluster-data/ts-1/data/info.pb
I20260926 03:46:07.250579 779 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu as pid 12320
I20260926 03:46:07.250792 779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790394221564982-779-0/minicluster-data/ts-1/wal/instance
I20260926 03:46:07.251394 779 external_mini_cluster.cc:1428] Setting key 1b24aaa91145e028059cea3a2d101543
Sep 26 03:46:07 dist-test-slave-n2th krb5kdc[11625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394367, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.194@KRBTEST.COM for kudu/127.0.194.254@KRBTEST.COM
I20260926 03:46:07.276677 12460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:46:07.253250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.194:46251 (local address 127.0.194.254:42187)
0926 03:46:07.254262 (+ 1012us) server_negotiation.cc:206] Beginning negotiation
0926 03:46:07.254268 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:46:07.254288 (+ 20us) server_negotiation.cc:407] Connection header received
0926 03:46:07.257155 (+ 2867us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:46:07.257161 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:46:07.257261 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:46:07.257374 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:46:07.259562 (+ 2188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:46:07.260583 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:46:07.262866 (+ 2283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:46:07.263581 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:46:07.263739 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:46:07.269039 (+ 5300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:46:07.269049 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:46:07.269058 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:46:07.269107 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:46:07.272894 (+ 3787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:46:07.273824 (+ 930us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:46:07.273832 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:46:07.273837 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:46:07.273991 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:46:07.274493 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:46:07.274499 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:46:07.274502 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:46:07.274905 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:46:07.275020 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0926 03:46:07.275924 (+ 904us) server_negotiation.cc:299] Negotiation successful
0926 03:46:07.276201 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":894,"thread_start_us":1519,"threads_started":1}
I20260926 03:46:07.277891 12459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:46:07.252729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:42187 (local address 127.0.194.194:46251)
0926 03:46:07.253598 (+ 869us) negotiation.cc:107] Waiting for socket to connect
0926 03:46:07.253666 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0926 03:46:07.256813 (+ 3147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:46:07.257698 (+ 885us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:46:07.257713 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:46:07.258397 (+ 684us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:46:07.259330 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:46:07.259360 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:46:07.260784 (+ 1424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:46:07.260789 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:46:07.262672 (+ 1883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:46:07.262686 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:46:07.263835 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:46:07.263841 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:46:07.264110 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:46:07.265942 (+ 1832us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:46:07.265971 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:46:07.268731 (+ 2760us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:46:07.273071 (+ 4340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:46:07.273081 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:46:07.273100 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:46:07.273664 (+ 564us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:46:07.274107 (+ 443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:46:07.274112 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:46:07.274115 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:46:07.274360 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:46:07.275018 (+ 658us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:46:07.275025 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:46:07.275465 (+ 440us) client_negotiation.cc:769] Sending connection context
0926 03:46:07.275754 (+ 289us) client_negotiation.cc:240] Negotiation successful
0926 03:46:07.276288 (+ 534us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":571,"thread_start_us":319,"threads_started":1}
WARNING: no policy specified for kudu/127.0.194.195@KRBTEST.COM; defaulting to no policy
I20260926 03:46:07.279482 12457 heartbeater.cc:347] Connected to a master server at 127.0.194.254:42187
I20260926 03:46:07.279819 12457 heartbeater.cc:464] Registering TS with master...
I20260926 03:46:07.280529 12457 heartbeater.cc:511] Master 127.0.194.254:42187 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.194.195@KRBTEST.COM" created.
I20260926 03:46:07.282182 12101 ts_manager.cc:194] Registered new tserver with Master: c17a512c73c942db9c14966832291317 (127.0.194.194:37315)
I20260926 03:46:07.283296 12101 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.194.194@KRBTEST.COM'} at 127.0.194.194:46251
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.194.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790394221564982-779-0/krb5kdc/kudu_127.0.194.195.keytab.
Entry for principal kudu/127.0.194.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790394221564982-779-0/krb5kdc/kudu_127.0.194.195.keytab.
I20260926 03:46:07.310974 779 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
/tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790394221564982-779-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790394221564982-779-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790394221564982-779-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790394221564982-779-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.194.212:48877/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.194.195:0
--local_ip_for_outbound_sockets=127.0.194.195
--webserver_interface=127.0.194.195
--webserver_port=0
--tserver_master_addrs=127.0.194.254:42187
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790394221564982-779-0/krb5kdc/kudu_127.0.194.195.keytab
--principal=kudu/127.0.194.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.194.212:37757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790394221564982-779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790394221564982-779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790394221564982-779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 03:46:07.425849 12314 heartbeater.cc:503] Master 127.0.194.254:42187 was elected leader, sending a full tablet report...
W20260926 03:46:07.661247 12463 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:46:07.661616 12463 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:46:07.661800 12463 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:46:07.669704 12463 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:46:07.669831 12463 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:46:07.670033 12463 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.194.195
I20260926 03:46:07.680343 12463 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.194.212:37757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790394221564982-779-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790394221564982-779-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.194.212:48877/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790394221564982-779-0/krb5kdc/kudu_127.0.194.195.keytab
--principal=kudu/127.0.194.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.194.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790394221564982-779-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.194.195
--webserver_port=0
--tserver_master_addrs=127.0.194.254:42187
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.194.195
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790394221564982-779-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:46:07.681701 12463 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:46:07.683147 12463 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:46:07.697356 12468 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:46:07.697402 12469 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:46:07.697451 12471 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:46:07.700093 12463 server_base.cc:1034] running on GCE node
I20260926 03:46:07.700835 12463 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:46:07.701877 12463 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:46:07.703078 12463 hybrid_clock.cc:648] HybridClock initialized: now 1790394367703022 us; error 48 us; skew 500 ppm
I20260926 03:46:07.703419 12463 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:46:07 dist-test-slave-n2th krb5kdc[11625](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394367, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:46:07.706503 12463 init.cc:377] Logged in from keytab as kudu/127.0.194.195@KRBTEST.COM (short username kudu)
I20260926 03:46:07.708685 12463 webserver.cc:460] Webserver started at http://127.0.194.195:33929/ using document root <none> and password file <none>
I20260926 03:46:07.709438 12463 fs_manager.cc:359] Metadata directory not provided
I20260926 03:46:07.709589 12463 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:46:07.709911 12463 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 03:46:07 dist-test-slave-n2th krb5kdc[11625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394367, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.195@KRBTEST.COM for HTTP/127.0.194.212@KRBTEST.COM
I20260926 03:46:07.727964 12463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790394221564982-779-0/minicluster-data/ts-2/data/instance:
uuid: "bdae5dab336547a299d40c4bed0d257b"
format_stamp: "Formatted at 2026-09-26 03:46:07 on dist-test-slave-n2th"
server_key: "796f15783bc1812177a79c744f9dc080"
server_key_iv: "5a1b911feffd5005dec6ab0d0bcf876b"
server_key_version: "kuduclusterkey@0"
I20260926 03:46:07.728868 12463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790394221564982-779-0/minicluster-data/ts-2/wal/instance:
uuid: "bdae5dab336547a299d40c4bed0d257b"
format_stamp: "Formatted at 2026-09-26 03:46:07 on dist-test-slave-n2th"
server_key: "796f15783bc1812177a79c744f9dc080"
server_key_iv: "5a1b911feffd5005dec6ab0d0bcf876b"
server_key_version: "kuduclusterkey@0"
I20260926 03:46:07.734295 12463 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260926 03:46:07.749534 12478 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:46:07.751304 12463 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 03:46:07.751490 12463 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790394221564982-779-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790394221564982-779-0/minicluster-data/ts-2/wal
uuid: "bdae5dab336547a299d40c4bed0d257b"
format_stamp: "Formatted at 2026-09-26 03:46:07 on dist-test-slave-n2th"
server_key: "796f15783bc1812177a79c744f9dc080"
server_key_iv: "5a1b911feffd5005dec6ab0d0bcf876b"
server_key_version: "kuduclusterkey@0"
I20260926 03:46:07.751720 12463 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790394221564982-779-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790394221564982-779-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790394221564982-779-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:46:07.769415 12463 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:46:07.773672 12463 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:46:07.774011 12463 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:46:07.774999 12463 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:46:07.776311 12463 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:46:07.776383 12463 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:46:07.776547 12463 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:46:07.776605 12463 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:46:07.811957 12463 rpc_server.cc:307] RPC server started. Bound to: 127.0.194.195:33403
I20260926 03:46:07.811990 12590 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.194.195:33403 every 8 connection(s)
I20260926 03:46:07.813261 12463 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790394221564982-779-0/minicluster-data/ts-2/data/info.pb
I20260926 03:46:07.814960 779 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu as pid 12463
I20260926 03:46:07.815126 779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790394221564982-779-0/minicluster-data/ts-2/wal/instance
I20260926 03:46:07.815572 779 external_mini_cluster.cc:1428] Setting key 53453f5211ebab0b5d8db65e65b7eaaa
Sep 26 03:46:07 dist-test-slave-n2th krb5kdc[11625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394367, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.195@KRBTEST.COM for kudu/127.0.194.254@KRBTEST.COM
I20260926 03:46:07.833382 12593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:46:07.816255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.195:51037 (local address 127.0.194.254:42187)
0926 03:46:07.816813 (+ 558us) server_negotiation.cc:206] Beginning negotiation
0926 03:46:07.816821 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:46:07.817604 (+ 783us) server_negotiation.cc:407] Connection header received
0926 03:46:07.819500 (+ 1896us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:46:07.819505 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:46:07.819577 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:46:07.819666 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:46:07.821031 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:46:07.821738 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:46:07.823346 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:46:07.824174 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:46:07.824271 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:46:07.827514 (+ 3243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:46:07.827519 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:46:07.827522 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:46:07.827553 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:46:07.830186 (+ 2633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:46:07.831089 (+ 903us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:46:07.831098 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:46:07.831105 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:46:07.831232 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:46:07.831686 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:46:07.831695 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:46:07.831697 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:46:07.831946 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:46:07.832109 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0926 03:46:07.832792 (+ 683us) server_negotiation.cc:299] Negotiation successful
0926 03:46:07.833003 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":404,"thread_start_us":271,"threads_started":1}
I20260926 03:46:07.834340 12594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:46:07.816382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:42187 (local address 127.0.194.195:51037)
0926 03:46:07.817252 (+ 870us) negotiation.cc:107] Waiting for socket to connect
0926 03:46:07.817305 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 03:46:07.819258 (+ 1953us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:46:07.819874 (+ 616us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:46:07.819882 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:46:07.820310 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:46:07.820883 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:46:07.820907 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:46:07.821867 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:46:07.821871 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:46:07.823221 (+ 1350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:46:07.823232 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:46:07.824374 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:46:07.824379 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:46:07.824569 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:46:07.825190 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:46:07.825224 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:46:07.827281 (+ 2057us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:46:07.830394 (+ 3113us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:46:07.830404 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:46:07.830423 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:46:07.830963 (+ 540us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:46:07.831346 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:46:07.831352 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:46:07.831356 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:46:07.831561 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:46:07.832080 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:46:07.832089 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:46:07.832430 (+ 341us) client_negotiation.cc:769] Sending connection context
0926 03:46:07.832698 (+ 268us) client_negotiation.cc:240] Negotiation successful
0926 03:46:07.833238 (+ 540us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":551,"thread_start_us":350,"threads_started":1}
I20260926 03:46:07.835754 12591 heartbeater.cc:347] Connected to a master server at 127.0.194.254:42187
I20260926 03:46:07.836081 12591 heartbeater.cc:464] Registering TS with master...
I20260926 03:46:07.836818 12591 heartbeater.cc:511] Master 127.0.194.254:42187 requested a full tablet report, sending...
I20260926 03:46:07.838289 12101 ts_manager.cc:194] Registered new tserver with Master: bdae5dab336547a299d40c4bed0d257b (127.0.194.195:33403)
I20260926 03:46:07.839222 12101 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.194.195@KRBTEST.COM'} at 127.0.194.195:51037
I20260926 03:46:07.843331 779 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 03:46:07.861158 12593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:46:07.847829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60714 (local address 127.0.194.254:42187)
0926 03:46:07.848066 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0926 03:46:07.848075 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:46:07.848484 (+ 409us) server_negotiation.cc:407] Connection header received
0926 03:46:07.848691 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:46:07.848696 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:46:07.848779 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:46:07.848884 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:46:07.850241 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:46:07.851032 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:46:07.852378 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:46:07.853215 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:46:07.853356 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:46:07.855963 (+ 2607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:46:07.855968 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:46:07.855971 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:46:07.856006 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:46:07.858445 (+ 2439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:46:07.859228 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:46:07.859234 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:46:07.859238 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:46:07.859362 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:46:07.859751 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:46:07.859756 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:46:07.859760 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:46:07.860039 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:46:07.860279 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0926 03:46:07.860612 (+ 333us) server_negotiation.cc:299] Negotiation successful
0926 03:46:07.860839 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260926 03:46:07.865360 12101 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60714:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 03:46:07.872378 12101 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 03:46:07.899684 12605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:46:07.885508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55500 (local address 127.0.194.195:33403)
0926 03:46:07.888750 (+ 3242us) server_negotiation.cc:206] Beginning negotiation
0926 03:46:07.888764 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 03:46:07.888809 (+ 45us) server_negotiation.cc:407] Connection header received
0926 03:46:07.889366 (+ 557us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:46:07.889378 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:46:07.889692 (+ 314us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:46:07.890981 (+ 1289us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:46:07.891321 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:46:07.893741 (+ 2420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:46:07.897102 (+ 3361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:46:07.898405 (+ 1303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:46:07.898555 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:46:07.898820 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0926 03:46:07.899156 (+ 336us) server_negotiation.cc:299] Negotiation successful
0926 03:46:07.899314 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2991,"thread_start_us":2806,"threads_started":1}
I20260926 03:46:07.899813 12606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:46:07.885271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.195:33403 (local address 127.0.0.1:55500)
0926 03:46:07.887579 (+ 2308us) negotiation.cc:107] Waiting for socket to connect
0926 03:46:07.887650 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0926 03:46:07.887885 (+ 235us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:46:07.889966 (+ 2081us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:46:07.889974 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:46:07.890027 (+ 53us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:46:07.890618 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:46:07.890638 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:46:07.893943 (+ 3305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:46:07.893957 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:46:07.896873 (+ 2916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:46:07.896887 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:46:07.898597 (+ 1710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:46:07.898603 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:46:07.898832 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:46:07.898916 (+ 84us) client_negotiation.cc:769] Sending connection context
0926 03:46:07.899103 (+ 187us) client_negotiation.cc:240] Negotiation successful
0926 03:46:07.899309 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1934,"mutex_wait_us":33,"thread_start_us":3866,"threads_started":1}
I20260926 03:46:07.900521 12604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:46:07.886325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.193:32843 (local address 127.0.0.1:40072)
0926 03:46:07.889411 (+ 3086us) negotiation.cc:107] Waiting for socket to connect
0926 03:46:07.889428 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 03:46:07.889546 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:46:07.890477 (+ 931us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:46:07.890482 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:46:07.890510 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:46:07.890928 (+ 418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:46:07.890943 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:46:07.893145 (+ 2202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:46:07.893150 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:46:07.897070 (+ 3920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:46:07.897087 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:46:07.899548 (+ 2461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:46:07.899554 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:46:07.899825 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:46:07.899860 (+ 35us) client_negotiation.cc:769] Sending connection context
0926 03:46:07.899953 (+ 93us) client_negotiation.cc:240] Negotiation successful
0926 03:46:07.900096 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2968,"mutex_wait_us":5,"thread_start_us":873,"threads_started":1}
I20260926 03:46:07.903128 12607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:46:07.887033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40072 (local address 127.0.194.193:32843)
0926 03:46:07.889145 (+ 2112us) server_negotiation.cc:206] Beginning negotiation
0926 03:46:07.889156 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 03:46:07.889674 (+ 518us) server_negotiation.cc:407] Connection header received
0926 03:46:07.889777 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:46:07.889786 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:46:07.890014 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:46:07.890231 (+ 217us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:46:07.891123 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:46:07.892985 (+ 1862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:46:07.897282 (+ 4297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:46:07.899006 (+ 1724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:46:07.899134 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:46:07.899393 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0926 03:46:07.900785 (+ 1392us) server_negotiation.cc:299] Negotiation successful
0926 03:46:07.902875 (+ 2090us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1934,"thread_start_us":1774,"threads_started":1}
I20260926 03:46:07.904172 12609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:46:07.891050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.194:37315 (local address 127.0.0.1:32952)
0926 03:46:07.895391 (+ 4341us) negotiation.cc:107] Waiting for socket to connect
0926 03:46:07.895407 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 03:46:07.895579 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:46:07.896257 (+ 678us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:46:07.896263 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:46:07.896292 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:46:07.896771 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:46:07.896783 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:46:07.898952 (+ 2169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:46:07.898959 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:46:07.901414 (+ 2455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:46:07.901429 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:46:07.903427 (+ 1998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:46:07.903434 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:46:07.903609 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:46:07.903644 (+ 35us) client_negotiation.cc:769] Sending connection context
0926 03:46:07.903715 (+ 71us) client_negotiation.cc:240] Negotiation successful
0926 03:46:07.903846 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4236,"thread_start_us":4112,"threads_started":1}
I20260926 03:46:07.904330 12526 tablet_service.cc:1467] Processing CreateTablet for tablet 87c946f076594c669ad8d035ba1341ae (DEFAULT_TABLE table=test-table [id=a0c41ac4d69546809bc48cea06ec57b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:46:07.905830 12608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:46:07.889858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32952 (local address 127.0.194.194:37315)
0926 03:46:07.895090 (+ 5232us) server_negotiation.cc:206] Beginning negotiation
0926 03:46:07.895100 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 03:46:07.895531 (+ 431us) server_negotiation.cc:407] Connection header received
0926 03:46:07.895780 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:46:07.895789 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:46:07.896033 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:46:07.896244 (+ 211us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:46:07.897019 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:46:07.898779 (+ 1760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:46:07.901629 (+ 2850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:46:07.903173 (+ 1544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:46:07.904548 (+ 1375us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:46:07.904755 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0926 03:46:07.904824 (+ 69us) server_negotiation.cc:299] Negotiation successful
0926 03:46:07.904991 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5036,"thread_start_us":4918,"threads_started":1}
I20260926 03:46:07.906476 12526 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87c946f076594c669ad8d035ba1341ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:46:07.906181 12249 tablet_service.cc:1467] Processing CreateTablet for tablet 87c946f076594c669ad8d035ba1341ae (DEFAULT_TABLE table=test-table [id=a0c41ac4d69546809bc48cea06ec57b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:46:07.907567 12249 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87c946f076594c669ad8d035ba1341ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:46:07.909351 12392 tablet_service.cc:1467] Processing CreateTablet for tablet 87c946f076594c669ad8d035ba1341ae (DEFAULT_TABLE table=test-table [id=a0c41ac4d69546809bc48cea06ec57b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:46:07.910673 12392 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87c946f076594c669ad8d035ba1341ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:46:07.919396 12610 tablet_bootstrap.cc:492] T 87c946f076594c669ad8d035ba1341ae P bdae5dab336547a299d40c4bed0d257b: Bootstrap starting.
I20260926 03:46:07.922060 12611 tablet_bootstrap.cc:492] T 87c946f076594c669ad8d035ba1341ae P d76665909ce842df9635c0f625f7c795: Bootstrap starting.
I20260926 03:46:07.923321 12610 tablet_bootstrap.cc:654] T 87c946f076594c669ad8d035ba1341ae P bdae5dab336547a299d40c4bed0d257b: Neither blocks nor log segments found. Creating new log.
I20260926 03:46:07.924659 12612 tablet_bootstrap.cc:492] T 87c946f076594c669ad8d035ba1341ae P c17a512c73c942db9c14966832291317: Bootstrap starting.
I20260926 03:46:07.926416 12611 tablet_bootstrap.cc:654] T 87c946f076594c669ad8d035ba1341ae P d76665909ce842df9635c0f625f7c795: Neither blocks nor log segments found. Creating new log.
I20260926 03:46:07.926587 12610 log.cc:824] T 87c946f076594c669ad8d035ba1341ae P bdae5dab336547a299d40c4bed0d257b: Log is configured to *not* fsync() on all Append() calls
I20260926 03:46:07.927659 12612 tablet_bootstrap.cc:654] T 87c946f076594c669ad8d035ba1341ae P c17a512c73c942db9c14966832291317: Neither blocks nor log segments found. Creating new log.
I20260926 03:46:07.927829 12611 log.cc:824] T 87c946f076594c669ad8d035ba1341ae P d76665909ce842df9635c0f625f7c795: Log is configured to *not* fsync() on all Append() calls
I20260926 03:46:07.929332 12612 log.cc:824] T 87c946f076594c669ad8d035ba1341ae P c17a512c73c942db9c14966832291317: Log is configured to *not* fsync() on all Append() calls
I20260926 03:46:07.929986 12610 tablet_bootstrap.cc:492] T 87c946f076594c669ad8d035ba1341ae P bdae5dab336547a299d40c4bed0d257b: No bootstrap required, opened a new log
I20260926 03:46:07.930310 12610 ts_tablet_manager.cc:1397] T 87c946f076594c669ad8d035ba1341ae P bdae5dab336547a299d40c4bed0d257b: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.002s
I20260926 03:46:07.931125 12611 tablet_bootstrap.cc:492] T 87c946f076594c669ad8d035ba1341ae P d76665909ce842df9635c0f625f7c795: No bootstrap required, opened a new log
I20260926 03:46:07.931403 12611 ts_tablet_manager.cc:1397] T 87c946f076594c669ad8d035ba1341ae P d76665909ce842df9635c0f625f7c795: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260926 03:46:07.931849 12612 tablet_bootstrap.cc:492] T 87c946f076594c669ad8d035ba1341ae P c17a512c73c942db9c14966832291317: No bootstrap required, opened a new log
I20260926 03:46:07.932111 12612 ts_tablet_manager.cc:1397] T 87c946f076594c669ad8d035ba1341ae P c17a512c73c942db9c14966832291317: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260926 03:46:07.939752 12612 raft_consensus.cc:348] T 87c946f076594c669ad8d035ba1341ae P c17a512c73c942db9c14966832291317 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdae5dab336547a299d40c4bed0d257b" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 33403 } } peers { permanent_uuid: "c17a512c73c942db9c14966832291317" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 37315 } } peers { permanent_uuid: "d76665909ce842df9635c0f625f7c795" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 32843 } }
I20260926 03:46:07.940145 12612 raft_consensus.cc:374] T 87c946f076594c669ad8d035ba1341ae P c17a512c73c942db9c14966832291317 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:46:07.940265 12612 raft_consensus.cc:729] T 87c946f076594c669ad8d035ba1341ae P c17a512c73c942db9c14966832291317 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c17a512c73c942db9c14966832291317, State: Initialized, Role: FOLLOWER
I20260926 03:46:07.940886 12612 consensus_queue.cc:260] T 87c946f076594c669ad8d035ba1341ae P c17a512c73c942db9c14966832291317 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdae5dab336547a299d40c4bed0d257b" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 33403 } } peers { permanent_uuid: "c17a512c73c942db9c14966832291317" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 37315 } } peers { permanent_uuid: "d76665909ce842df9635c0f625f7c795" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 32843 } }
I20260926 03:46:07.942296 12457 heartbeater.cc:503] Master 127.0.194.254:42187 was elected leader, sending a full tablet report...
I20260926 03:46:07.942687 12610 raft_consensus.cc:348] T 87c946f076594c669ad8d035ba1341ae P bdae5dab336547a299d40c4bed0d257b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdae5dab336547a299d40c4bed0d257b" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 33403 } } peers { permanent_uuid: "c17a512c73c942db9c14966832291317" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 37315 } } peers { permanent_uuid: "d76665909ce842df9635c0f625f7c795" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 32843 } }
I20260926 03:46:07.943173 12610 raft_consensus.cc:374] T 87c946f076594c669ad8d035ba1341ae P bdae5dab336547a299d40c4bed0d257b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:46:07.943290 12610 raft_consensus.cc:729] T 87c946f076594c669ad8d035ba1341ae P bdae5dab336547a299d40c4bed0d257b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bdae5dab336547a299d40c4bed0d257b, State: Initialized, Role: FOLLOWER
I20260926 03:46:07.943691 12611 raft_consensus.cc:348] T 87c946f076594c669ad8d035ba1341ae P d76665909ce842df9635c0f625f7c795 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdae5dab336547a299d40c4bed0d257b" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 33403 } } peers { permanent_uuid: "c17a512c73c942db9c14966832291317" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 37315 } } peers { permanent_uuid: "d76665909ce842df9635c0f625f7c795" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 32843 } }
I20260926 03:46:07.944147 12612 ts_tablet_manager.cc:1428] T 87c946f076594c669ad8d035ba1341ae P c17a512c73c942db9c14966832291317: Time spent starting tablet: real 0.012s user 0.011s sys 0.000s
I20260926 03:46:07.944213 12611 raft_consensus.cc:374] T 87c946f076594c669ad8d035ba1341ae P d76665909ce842df9635c0f625f7c795 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:46:07.944125 12610 consensus_queue.cc:260] T 87c946f076594c669ad8d035ba1341ae P bdae5dab336547a299d40c4bed0d257b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdae5dab336547a299d40c4bed0d257b" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 33403 } } peers { permanent_uuid: "c17a512c73c942db9c14966832291317" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 37315 } } peers { permanent_uuid: "d76665909ce842df9635c0f625f7c795" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 32843 } }
I20260926 03:46:07.944375 12611 raft_consensus.cc:729] T 87c946f076594c669ad8d035ba1341ae P d76665909ce842df9635c0f625f7c795 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d76665909ce842df9635c0f625f7c795, State: Initialized, Role: FOLLOWER
I20260926 03:46:07.945338 12591 heartbeater.cc:503] Master 127.0.194.254:42187 was elected leader, sending a full tablet report...
I20260926 03:46:07.946213 12610 ts_tablet_manager.cc:1428] T 87c946f076594c669ad8d035ba1341ae P bdae5dab336547a299d40c4bed0d257b: Time spent starting tablet: real 0.016s user 0.016s sys 0.001s
I20260926 03:46:07.946745 12611 consensus_queue.cc:260] T 87c946f076594c669ad8d035ba1341ae P d76665909ce842df9635c0f625f7c795 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdae5dab336547a299d40c4bed0d257b" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 33403 } } peers { permanent_uuid: "c17a512c73c942db9c14966832291317" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 37315 } } peers { permanent_uuid: "d76665909ce842df9635c0f625f7c795" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 32843 } }
I20260926 03:46:07.949947 12611 ts_tablet_manager.cc:1428] T 87c946f076594c669ad8d035ba1341ae P d76665909ce842df9635c0f625f7c795: Time spent starting tablet: real 0.018s user 0.016s sys 0.000s
W20260926 03:46:08.001071 12458 tablet.cc:2369] T 87c946f076594c669ad8d035ba1341ae P c17a512c73c942db9c14966832291317: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 03:46:08.065521 12592 tablet.cc:2369] T 87c946f076594c669ad8d035ba1341ae P bdae5dab336547a299d40c4bed0d257b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 03:46:08.141309 12318 tablet.cc:2369] T 87c946f076594c669ad8d035ba1341ae P d76665909ce842df9635c0f625f7c795: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 03:46:08.255981 12617 raft_consensus.cc:482] T 87c946f076594c669ad8d035ba1341ae P bdae5dab336547a299d40c4bed0d257b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 03:46:08.256326 12617 raft_consensus.cc:504] T 87c946f076594c669ad8d035ba1341ae P bdae5dab336547a299d40c4bed0d257b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdae5dab336547a299d40c4bed0d257b" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 33403 } } peers { permanent_uuid: "c17a512c73c942db9c14966832291317" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 37315 } } peers { permanent_uuid: "d76665909ce842df9635c0f625f7c795" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 32843 } }
I20260926 03:46:08.257982 12617 leader_election.cc:288] T 87c946f076594c669ad8d035ba1341ae P bdae5dab336547a299d40c4bed0d257b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c17a512c73c942db9c14966832291317 (127.0.194.194:37315), d76665909ce842df9635c0f625f7c795 (127.0.194.193:32843)
I20260926 03:46:08.263605 12608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:46:08.258308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.195:59025 (local address 127.0.194.194:37315)
0926 03:46:08.258554 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0926 03:46:08.258562 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:46:08.259018 (+ 456us) server_negotiation.cc:407] Connection header received
0926 03:46:08.259077 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:46:08.259081 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:46:08.259136 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:46:08.259233 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:46:08.259596 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:46:08.260304 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:46:08.261912 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:46:08.262665 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:46:08.262869 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:46:08.262944 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 03:46:08.263238 (+ 294us) server_negotiation.cc:299] Negotiation successful
0926 03:46:08.263357 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260926 03:46:08.263626 12594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:46:08.258409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.194:37315 (local address 127.0.194.195:59025)
0926 03:46:08.258730 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0926 03:46:08.258777 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 03:46:08.258906 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:46:08.259247 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:46:08.259251 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:46:08.259285 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:46:08.259464 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:46:08.259471 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:46:08.260431 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:46:08.260439 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:46:08.261755 (+ 1316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:46:08.261767 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:46:08.262859 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:46:08.262864 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:46:08.263037 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:46:08.263088 (+ 51us) client_negotiation.cc:769] Sending connection context
0926 03:46:08.263193 (+ 105us) client_negotiation.cc:240] Negotiation successful
0926 03:46:08.263329 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"spinlock_wait_cycles":7424}
I20260926 03:46:08.264387 12607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:46:08.258627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.195:60427 (local address 127.0.194.193:32843)
0926 03:46:08.258863 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0926 03:46:08.258869 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:46:08.259246 (+ 377us) server_negotiation.cc:407] Connection header received
0926 03:46:08.259471 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:46:08.259482 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:46:08.259539 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:46:08.259638 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:46:08.260126 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:46:08.260833 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:46:08.262446 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:46:08.263418 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:46:08.263596 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:46:08.263664 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 03:46:08.264039 (+ 375us) server_negotiation.cc:299] Negotiation successful
0926 03:46:08.264186 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 03:46:08.264493 12619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:46:08.258468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.193:32843 (local address 127.0.194.195:60427)
0926 03:46:08.259142 (+ 674us) negotiation.cc:107] Waiting for socket to connect
0926 03:46:08.259156 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 03:46:08.259329 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:46:08.259649 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:46:08.259654 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:46:08.259676 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:46:08.259979 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:46:08.259988 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:46:08.260982 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:46:08.260987 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:46:08.262293 (+ 1306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:46:08.262304 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:46:08.263618 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:46:08.263624 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:46:08.263833 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:46:08.263864 (+ 31us) client_negotiation.cc:769] Sending connection context
0926 03:46:08.263981 (+ 117us) client_negotiation.cc:240] Negotiation successful
0926 03:46:08.264106 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":553,"spinlock_wait_cycles":27392,"thread_start_us":387,"threads_started":1}
I20260926 03:46:08.264848 12412 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87c946f076594c669ad8d035ba1341ae" candidate_uuid: "bdae5dab336547a299d40c4bed0d257b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c17a512c73c942db9c14966832291317" is_pre_election: true
I20260926 03:46:08.265436 12412 raft_consensus.cc:2454] T 87c946f076594c669ad8d035ba1341ae P c17a512c73c942db9c14966832291317 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bdae5dab336547a299d40c4bed0d257b in term 0.
I20260926 03:46:08.265667 12269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87c946f076594c669ad8d035ba1341ae" candidate_uuid: "bdae5dab336547a299d40c4bed0d257b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d76665909ce842df9635c0f625f7c795" is_pre_election: true
I20260926 03:46:08.266151 12480 leader_election.cc:302] T 87c946f076594c669ad8d035ba1341ae P bdae5dab336547a299d40c4bed0d257b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bdae5dab336547a299d40c4bed0d257b, c17a512c73c942db9c14966832291317; no voters:
I20260926 03:46:08.266193 12269 raft_consensus.cc:2454] T 87c946f076594c669ad8d035ba1341ae P d76665909ce842df9635c0f625f7c795 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bdae5dab336547a299d40c4bed0d257b in term 0.
I20260926 03:46:08.266535 12617 raft_consensus.cc:2781] T 87c946f076594c669ad8d035ba1341ae P bdae5dab336547a299d40c4bed0d257b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 03:46:08.266710 12617 raft_consensus.cc:482] T 87c946f076594c669ad8d035ba1341ae P bdae5dab336547a299d40c4bed0d257b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 03:46:08.266855 12617 raft_consensus.cc:3037] T 87c946f076594c669ad8d035ba1341ae P bdae5dab336547a299d40c4bed0d257b [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:46:08.269014 12617 raft_consensus.cc:504] T 87c946f076594c669ad8d035ba1341ae P bdae5dab336547a299d40c4bed0d257b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdae5dab336547a299d40c4bed0d257b" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 33403 } } peers { permanent_uuid: "c17a512c73c942db9c14966832291317" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 37315 } } peers { permanent_uuid: "d76665909ce842df9635c0f625f7c795" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 32843 } }
I20260926 03:46:08.269605 12617 leader_election.cc:288] T 87c946f076594c669ad8d035ba1341ae P bdae5dab336547a299d40c4bed0d257b [CANDIDATE]: Term 1 election: Requested vote from peers c17a512c73c942db9c14966832291317 (127.0.194.194:37315), d76665909ce842df9635c0f625f7c795 (127.0.194.193:32843)
I20260926 03:46:08.270118 12269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87c946f076594c669ad8d035ba1341ae" candidate_uuid: "bdae5dab336547a299d40c4bed0d257b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d76665909ce842df9635c0f625f7c795"
I20260926 03:46:08.270190 12412 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87c946f076594c669ad8d035ba1341ae" candidate_uuid: "bdae5dab336547a299d40c4bed0d257b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c17a512c73c942db9c14966832291317"
I20260926 03:46:08.270327 12269 raft_consensus.cc:3037] T 87c946f076594c669ad8d035ba1341ae P d76665909ce842df9635c0f625f7c795 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:46:08.270444 12412 raft_consensus.cc:3037] T 87c946f076594c669ad8d035ba1341ae P c17a512c73c942db9c14966832291317 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:46:08.273185 12269 raft_consensus.cc:2454] T 87c946f076594c669ad8d035ba1341ae P d76665909ce842df9635c0f625f7c795 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bdae5dab336547a299d40c4bed0d257b in term 1.
I20260926 03:46:08.273221 12412 raft_consensus.cc:2454] T 87c946f076594c669ad8d035ba1341ae P c17a512c73c942db9c14966832291317 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bdae5dab336547a299d40c4bed0d257b in term 1.
I20260926 03:46:08.273690 12481 leader_election.cc:302] T 87c946f076594c669ad8d035ba1341ae P bdae5dab336547a299d40c4bed0d257b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: bdae5dab336547a299d40c4bed0d257b, c17a512c73c942db9c14966832291317, d76665909ce842df9635c0f625f7c795; no voters:
I20260926 03:46:08.274022 12617 raft_consensus.cc:2781] T 87c946f076594c669ad8d035ba1341ae P bdae5dab336547a299d40c4bed0d257b [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:46:08.274519 12617 raft_consensus.cc:686] T 87c946f076594c669ad8d035ba1341ae P bdae5dab336547a299d40c4bed0d257b [term 1 LEADER]: Becoming Leader. State: Replica: bdae5dab336547a299d40c4bed0d257b, State: Running, Role: LEADER
I20260926 03:46:08.275230 12617 consensus_queue.cc:237] T 87c946f076594c669ad8d035ba1341ae P bdae5dab336547a299d40c4bed0d257b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdae5dab336547a299d40c4bed0d257b" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 33403 } } peers { permanent_uuid: "c17a512c73c942db9c14966832291317" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 37315 } } peers { permanent_uuid: "d76665909ce842df9635c0f625f7c795" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 32843 } }
I20260926 03:46:08.279704 12101 catalog_manager.cc:5495] T 87c946f076594c669ad8d035ba1341ae P bdae5dab336547a299d40c4bed0d257b reported cstate change: term changed from 0 to 1, leader changed from <none> to bdae5dab336547a299d40c4bed0d257b (127.0.194.195). New cstate: current_term: 1 leader_uuid: "bdae5dab336547a299d40c4bed0d257b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdae5dab336547a299d40c4bed0d257b" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 33403 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c17a512c73c942db9c14966832291317" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 37315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d76665909ce842df9635c0f625f7c795" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 32843 } health_report { overall_health: UNKNOWN } } }
I20260926 03:46:08.351982 12605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:46:08.346015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55508 (local address 127.0.194.195:33403)
0926 03:46:08.346233 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0926 03:46:08.346240 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:46:08.346375 (+ 135us) server_negotiation.cc:407] Connection header received
0926 03:46:08.346695 (+ 320us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:46:08.346698 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:46:08.346833 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:46:08.346989 (+ 156us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:46:08.347361 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:46:08.348076 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:46:08.349535 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:46:08.350104 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:46:08.350236 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:46:08.350846 (+ 610us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:46:08.351379 (+ 533us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:46:08.351542 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0926 03:46:08.351626 (+ 84us) server_negotiation.cc:299] Negotiation successful
0926 03:46:08.351706 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260926 03:46:08.361089 12269 raft_consensus.cc:1261] T 87c946f076594c669ad8d035ba1341ae P d76665909ce842df9635c0f625f7c795 [term 1 FOLLOWER]: Refusing update from remote peer bdae5dab336547a299d40c4bed0d257b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:46:08.361083 12412 raft_consensus.cc:1261] T 87c946f076594c669ad8d035ba1341ae P c17a512c73c942db9c14966832291317 [term 1 FOLLOWER]: Refusing update from remote peer bdae5dab336547a299d40c4bed0d257b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:46:08.362124 12620 consensus_queue.cc:1035] T 87c946f076594c669ad8d035ba1341ae P bdae5dab336547a299d40c4bed0d257b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c17a512c73c942db9c14966832291317" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 37315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:46:08.362604 12617 consensus_queue.cc:1035] T 87c946f076594c669ad8d035ba1341ae P bdae5dab336547a299d40c4bed0d257b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d76665909ce842df9635c0f625f7c795" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 32843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:46:08.393141 12101 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60714:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:46:08.393452 12101 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60714:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:46:08.397056 12101 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b047ae3c851b4fbd8262e06259a8c9b0: Sending DeleteTablet for 3 replicas of tablet 87c946f076594c669ad8d035ba1341ae
I20260926 03:46:08.398403 12392 tablet_service.cc:1514] Processing DeleteTablet for tablet 87c946f076594c669ad8d035ba1341ae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:46:08 UTC) from {username='kudu', principal='kudu/127.0.194.254@KRBTEST.COM'} at 127.0.0.1:32952
I20260926 03:46:08.398414 12526 tablet_service.cc:1514] Processing DeleteTablet for tablet 87c946f076594c669ad8d035ba1341ae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:46:08 UTC) from {username='kudu', principal='kudu/127.0.194.254@KRBTEST.COM'} at 127.0.0.1:55500
I20260926 03:46:08.398466 12249 tablet_service.cc:1514] Processing DeleteTablet for tablet 87c946f076594c669ad8d035ba1341ae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:46:08 UTC) from {username='kudu', principal='kudu/127.0.194.254@KRBTEST.COM'} at 127.0.0.1:40072
I20260926 03:46:08.400749 779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu pid 12181
I20260926 03:46:08.414093 12633 tablet_replica.cc:330] T 87c946f076594c669ad8d035ba1341ae P bdae5dab336547a299d40c4bed0d257b: stopping tablet replica
I20260926 03:46:08.414701 12633 raft_consensus.cc:2229] T 87c946f076594c669ad8d035ba1341ae P bdae5dab336547a299d40c4bed0d257b [term 1 LEADER]: Raft consensus shutting down.
I20260926 03:46:08.415428 12633 raft_consensus.cc:2258] T 87c946f076594c669ad8d035ba1341ae P bdae5dab336547a299d40c4bed0d257b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:46:08.416266 12635 tablet_replica.cc:330] T 87c946f076594c669ad8d035ba1341ae P c17a512c73c942db9c14966832291317: stopping tablet replica
I20260926 03:46:08.416803 12635 raft_consensus.cc:2229] T 87c946f076594c669ad8d035ba1341ae P c17a512c73c942db9c14966832291317 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:46:08.417380 12635 raft_consensus.cc:2258] T 87c946f076594c669ad8d035ba1341ae P c17a512c73c942db9c14966832291317 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:46:08.419744 12635 ts_tablet_manager.cc:1905] T 87c946f076594c669ad8d035ba1341ae P c17a512c73c942db9c14966832291317: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:46:08.420646 12634 tablet_replica.cc:330] T 87c946f076594c669ad8d035ba1341ae P d76665909ce842df9635c0f625f7c795: stopping tablet replica
I20260926 03:46:08.421133 12634 raft_consensus.cc:2229] T 87c946f076594c669ad8d035ba1341ae P d76665909ce842df9635c0f625f7c795 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:46:08.421722 12634 raft_consensus.cc:2258] T 87c946f076594c669ad8d035ba1341ae P d76665909ce842df9635c0f625f7c795 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:46:08.421792 12633 ts_tablet_manager.cc:1905] T 87c946f076594c669ad8d035ba1341ae P bdae5dab336547a299d40c4bed0d257b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:46:08.423776 12634 ts_tablet_manager.cc:1905] T 87c946f076594c669ad8d035ba1341ae P d76665909ce842df9635c0f625f7c795: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:46:08.427456 12635 ts_tablet_manager.cc:1918] T 87c946f076594c669ad8d035ba1341ae P c17a512c73c942db9c14966832291317: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:46:08.428107 12635 log.cc:1196] T 87c946f076594c669ad8d035ba1341ae P c17a512c73c942db9c14966832291317: Deleting WAL directory at /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790394221564982-779-0/minicluster-data/ts-1/wal/wals/87c946f076594c669ad8d035ba1341ae
I20260926 03:46:08.428700 12635 ts_tablet_manager.cc:1939] T 87c946f076594c669ad8d035ba1341ae P c17a512c73c942db9c14966832291317: Deleting consensus metadata
Sep 26 03:46:08 dist-test-slave-n2th krb5kdc[11625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394364, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.194.193@KRBTEST.COM
I20260926 03:46:08.429594 12633 ts_tablet_manager.cc:1918] T 87c946f076594c669ad8d035ba1341ae P bdae5dab336547a299d40c4bed0d257b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:46:08.429808 12633 log.cc:1196] T 87c946f076594c669ad8d035ba1341ae P bdae5dab336547a299d40c4bed0d257b: Deleting WAL directory at /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790394221564982-779-0/minicluster-data/ts-2/wal/wals/87c946f076594c669ad8d035ba1341ae
I20260926 03:46:08.430330 12633 ts_tablet_manager.cc:1939] T 87c946f076594c669ad8d035ba1341ae P bdae5dab336547a299d40c4bed0d257b: Deleting consensus metadata
I20260926 03:46:08.430785 12086 catalog_manager.cc:4841] TS c17a512c73c942db9c14966832291317 (127.0.194.194:37315): tablet 87c946f076594c669ad8d035ba1341ae (table test-table [id=a0c41ac4d69546809bc48cea06ec57b6]) successfully deleted
I20260926 03:46:08.431049 12634 ts_tablet_manager.cc:1918] T 87c946f076594c669ad8d035ba1341ae P d76665909ce842df9635c0f625f7c795: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:46:08.431267 12634 log.cc:1196] T 87c946f076594c669ad8d035ba1341ae P d76665909ce842df9635c0f625f7c795: Deleting WAL directory at /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790394221564982-779-0/minicluster-data/ts-0/wal/wals/87c946f076594c669ad8d035ba1341ae
I20260926 03:46:08.432052 12086 catalog_manager.cc:4841] TS bdae5dab336547a299d40c4bed0d257b (127.0.194.195:33403): tablet 87c946f076594c669ad8d035ba1341ae (table test-table [id=a0c41ac4d69546809bc48cea06ec57b6]) successfully deleted
I20260926 03:46:08.432257 12634 ts_tablet_manager.cc:1939] T 87c946f076594c669ad8d035ba1341ae P d76665909ce842df9635c0f625f7c795: Deleting consensus metadata
I20260926 03:46:08.434103 12087 catalog_manager.cc:4841] TS d76665909ce842df9635c0f625f7c795 (127.0.194.193:32843): tablet 87c946f076594c669ad8d035ba1341ae (table test-table [id=a0c41ac4d69546809bc48cea06ec57b6]) successfully deleted
I20260926 03:46:08.437737 12607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:46:08.402514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40080 (local address 127.0.194.193:32843)
0926 03:46:08.403005 (+ 491us) server_negotiation.cc:206] Beginning negotiation
0926 03:46:08.403012 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:46:08.413203 (+ 10191us) server_negotiation.cc:407] Connection header received
0926 03:46:08.413422 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:46:08.413428 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:46:08.413512 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:46:08.413630 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:46:08.415850 (+ 2220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:46:08.416989 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:46:08.419007 (+ 2018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:46:08.419724 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:46:08.419847 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:46:08.430967 (+ 11120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:46:08.430982 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:46:08.430999 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:46:08.431070 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:46:08.434870 (+ 3800us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:46:08.435762 (+ 892us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:46:08.435769 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:46:08.435772 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:46:08.435890 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:46:08.436306 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:46:08.436308 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:46:08.436310 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:46:08.436624 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:46:08.436838 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0926 03:46:08.437265 (+ 427us) server_negotiation.cc:299] Negotiation successful
0926 03:46:08.437424 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356}
I20260926 03:46:08.438366 12309 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:46:08.533491 779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu with pid 12181
I20260926 03:46:08.551184 779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu pid 12320
Sep 26 03:46:08 dist-test-slave-n2th krb5kdc[11625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394364, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.194.194@KRBTEST.COM
I20260926 03:46:08.570428 12608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:46:08.551983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32964 (local address 127.0.194.194:37315)
0926 03:46:08.552179 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0926 03:46:08.552184 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:46:08.552324 (+ 140us) server_negotiation.cc:407] Connection header received
0926 03:46:08.552513 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:46:08.552517 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:46:08.552585 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:46:08.552669 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:46:08.554436 (+ 1767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:46:08.555372 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:46:08.557328 (+ 1956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:46:08.557749 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:46:08.557860 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:46:08.563534 (+ 5674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:46:08.563554 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:46:08.563571 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:46:08.563650 (+ 79us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:46:08.567084 (+ 3434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:46:08.567982 (+ 898us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:46:08.567991 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:46:08.567994 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:46:08.568147 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:46:08.568613 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:46:08.568619 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:46:08.568625 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:46:08.569032 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:46:08.569362 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0926 03:46:08.569752 (+ 390us) server_negotiation.cc:299] Negotiation successful
0926 03:46:08.570003 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260926 03:46:08.571278 12452 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:46:08.680413 779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu with pid 12320
I20260926 03:46:08.702315 779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu pid 12463
Sep 26 03:46:08 dist-test-slave-n2th krb5kdc[11625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394364, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.194.195@KRBTEST.COM
I20260926 03:46:08.722223 12605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:46:08.703187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55516 (local address 127.0.194.195:33403)
0926 03:46:08.703386 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0926 03:46:08.703392 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:46:08.703415 (+ 23us) server_negotiation.cc:407] Connection header received
0926 03:46:08.703556 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:46:08.703562 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:46:08.703653 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:46:08.703800 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:46:08.705110 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:46:08.706315 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:46:08.708116 (+ 1801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:46:08.708804 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:46:08.708945 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:46:08.714316 (+ 5371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:46:08.714334 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:46:08.714355 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:46:08.714414 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:46:08.718704 (+ 4290us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:46:08.719731 (+ 1027us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:46:08.719743 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:46:08.719746 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:46:08.719987 (+ 241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:46:08.720455 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:46:08.720461 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:46:08.720464 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:46:08.720962 (+ 498us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:46:08.721284 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0926 03:46:08.721592 (+ 308us) server_negotiation.cc:299] Negotiation successful
0926 03:46:08.721848 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260926 03:46:08.723126 12586 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:46:08.857034 779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu with pid 12463
I20260926 03:46:08.876341 779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu pid 12069
I20260926 03:46:08.877511 12131 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:46:08.988451 779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu with pid 12069
I20260926 03:46:09.010521 779 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260926 03:46:09.346941 779 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260926 03:46:09.347198 779 mini_ranger.cc:66] Stopping Ranger...
I20260926 03:46:09.693255 779 mini_ranger.cc:68] Stopped Ranger
2026-09-26 03:46:09.693 UTC [11655] LOG: received smart shutdown request
2026-09-26 03:46:09.697 UTC [11655] LOG: background worker "logical replication launcher" (PID 11664) exited with exit code 1
2026-09-26 03:46:09.697 UTC [11659] LOG: shutting down
2026-09-26 03:46:09.864 UTC [11655] LOG: database system is shut down
2026-09-26T03:46:09Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (50275 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:46:10 dist-test-slave-n2th krb5kdc[12649](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:46:10 dist-test-slave-n2th krb5kdc[12649](info): set up 2 sockets
Sep 26 03:46:10 dist-test-slave-n2th krb5kdc[12649](info): commencing operation
krb5kdc: starting...
W20260926 03:46:11.103209 779 mini_kdc.cc:121] Time spent starting KDC: real 1.077s user 0.005s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/krb5kdc/test-user.keytab.
Sep 26 03:46:11 dist-test-slave-n2th krb5kdc[12649](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394371, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:46:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:46:11Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.194.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.194.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.194.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/krb5kdc/rangeradmin_127.0.194.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.194.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/krb5kdc/rangeradmin_127.0.194.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.194.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.194.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.194.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/krb5kdc/rangerlookup_127.0.194.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.194.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/krb5kdc/rangerlookup_127.0.194.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.194.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.194.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.194.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/krb5kdc/HTTP_127.0.194.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.194.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/krb5kdc/HTTP_127.0.194.212@KRBTEST.COM.keytab.
I20260926 03:46:11.437045 779 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-tasktGRWtf/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/minicluster-data/postgres -l logfile start
2026-09-26 03:46:13.905 UTC [12679] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-26 03:46:13.906 UTC [12679] LOG: listening on IPv4 address "127.0.194.212", port 36559
2026-09-26 03:46:13.909 UTC [12679] LOG: listening on Unix socket "/tmp/.s.PGSQL.36559"
2026-09-26 03:46:13.930 UTC [12682] LOG: database system was shut down at 2026-09-26 03:46:12 UTC
2026-09-26 03:46:13.933 UTC [12679] LOG: database system is ready to accept connections
I20260926 03:46:14.922850 779 mini_postgres.cc:96] Postgres bound to 36559
2026-09-26 03:46:14.940 UTC [12690] FATAL: database "slave" does not exist
127.0.194.212:36559 - accepting connections
I20260926 03:46:14.941103 779 mini_ranger.cc:161] Starting Ranger...
I20260926 03:46:14.967837 779 mini_ranger.cc:84] Created miniranger Postgres user
I20260926 03:46:15.485739 779 mini_ranger.cc:87] Created ranger Postgres database
I20260926 03:46:15.485859 779 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/minicluster-data/ranger-admin
2026-09-26 03:46:15,780 [I] DB FLAVOR :POSTGRES
2026-09-26 03:46:15,781 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 03:46:15,781 [I] Checking connection..
2026-09-26 03:46:15,781 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktGRWtf/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktGRWtf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.194.212:36559/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-26 03:46:16,021 [I] Checking connection passed.
2026-09-26 03:46:16,022 [I] --------- Verifying version history table ---------
2026-09-26 03:46:16,022 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktGRWtf/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktGRWtf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.194.212:36559/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 03:46:16,285 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 03:46:16,285 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktGRWtf/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktGRWtf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.194.212:36559/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 03:46:16,543 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 03:46:16,543 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-26 03:46:16,543 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktGRWtf/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktGRWtf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.194.212:36559/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasktGRWtf/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-26 03:46:16.806 UTC [12754] WARNING: there is no transaction in progress
2026-09-26 03:46:16,813 [I] create_dbversion_catalog.sql file imported successfully
2026-09-26 03:46:16,813 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktGRWtf/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktGRWtf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.194.212:36559/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 03:46:17,077 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-26 03:46:17,077 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 03:46:17,077 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktGRWtf/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktGRWtf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.194.212:36559/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-26 03:46:17,329 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktGRWtf/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktGRWtf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.194.212:36559/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-26 03:46:17,565 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktGRWtf/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktGRWtf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.194.212:36559/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-n2th.c.gcp-upstream.internal','N') ;"
2026-09-26 03:46:17,827 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktGRWtf/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktGRWtf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.194.212:36559/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 03:46:18,092 [I] Table x_portal_user does not exist in database ranger
2026-09-26 03:46:18,092 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktGRWtf/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktGRWtf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.194.212:36559/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 03:46:18,339 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-26 03:46:18,339 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktGRWtf/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktGRWtf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.194.212:36559/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 03:46:18,584 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktGRWtf/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktGRWtf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.194.212:36559/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 03:46:18,837 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-26 03:46:18,838 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktGRWtf/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktGRWtf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.194.212:36559/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasktGRWtf/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-26 03:46:19.790 UTC [12862] WARNING: there is no transaction in progress
2026-09-26 03:46:19.806 UTC [12862] WARNING: there is no transaction in progress
2026-09-26 03:46:19.822 UTC [12862] WARNING: there is no transaction in progress
2026-09-26 03:46:19.838 UTC [12862] WARNING: there is no transaction in progress
2026-09-26 03:46:19.858 UTC [12862] WARNING: there is no transaction in progress
2026-09-26 03:46:19.879 UTC [12862] WARNING: there is no transaction in progress
2026-09-26 03:46:19.936 UTC [12862] WARNING: there is no transaction in progress
2026-09-26 03:46:19.942 UTC [12862] WARNING: there is no transaction in progress
2026-09-26 03:46:19.957 UTC [12862] WARNING: there is no transaction in progress
2026-09-26 03:46:19.967 UTC [12862] WARNING: there is no transaction in progress
2026-09-26 03:46:19.977 UTC [12862] WARNING: there is no transaction in progress
2026-09-26 03:46:19.990 UTC [12862] WARNING: there is no transaction in progress
2026-09-26 03:46:20.000 UTC [12862] WARNING: there is no transaction in progress
2026-09-26 03:46:20.006 UTC [12862] WARNING: there is no transaction in progress
2026-09-26 03:46:20.013 UTC [12862] WARNING: there is no transaction in progress
2026-09-26 03:46:20.020 UTC [12862] WARNING: there is no transaction in progress
2026-09-26 03:46:20,629 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-26 03:46:20,630 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktGRWtf/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktGRWtf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.194.212:36559/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-26 03:46:20,900 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-26 03:46:20,901 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktGRWtf/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktGRWtf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.194.212:36559/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 03:46:21,170 [I] Table x_portal_user already exists in database 'ranger'
2026-09-26 03:46:21,170 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktGRWtf/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktGRWtf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.194.212:36559/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 03:46:21,437 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-26 03:46:21,437 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktGRWtf/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktGRWtf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.194.212:36559/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 03:46:21,673 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktGRWtf/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktGRWtf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.194.212:36559/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 03:46:21,933 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktGRWtf/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktGRWtf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.194.212:36559/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-n2th.c.gcp-upstream.internal';"
2026-09-26 03:46:22,194 [I] CORE_DB_SCHEMA import status has been updated
2026-09-26 03:46:22,195 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktGRWtf/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktGRWtf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.194.212:36559/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-26 03:46:22,445 [I] DB_PATCHES have already been applied
I20260926 03:46:22.451764 779 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/minicluster-data/ranger-admin:/tmp/dist-test-tasktGRWtf/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-tasktGRWtf/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-tasktGRWtf/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktGRWtf/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260926 03:46:22.451864 779 mini_ranger.cc:193] Using host: 127.0.194.212
I20260926 03:46:22.467001 779 mini_ranger.cc:227] Ranger admin URL: http://127.0.194.212:59041
Sep 26, 2026 3:46:22 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 3:46:22 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 3:46:22 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/minicluster-data/ranger-admin/ews/webapp
Sep 26, 2026 3:46:22 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 26, 2026 3:46:22 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 26, 2026 3:46:23 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/minicluster-data/ranger-admin/ews/webapp].
Sep 26, 2026 3:46:23 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260926 03:46:23.619292 779 mini_ranger.cc:160] Time spent starting Ranger: real 8.678s user 0.005s sys 0.036s
Sep 26, 2026 3:46:44 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 26, 2026 3:46:44 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.XKeyREST
Sep 26, 2026 3:46:44 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 26, 2026 3:46:44 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 26, 2026 3:46:44 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 26, 2026 3:46:44 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 26, 2026 3:46:44 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 26, 2026 3:46:44 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 26, 2026 3:46:44 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 26, 2026 3:46:44 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 26, 2026 3:46:44 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 26, 2026 3:46:44 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 26, 2026 3:46:44 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 26, 2026 3:46:44 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 26, 2026 3:46:44 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 26, 2026 3:46:44 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 26, 2026 3:46:44 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 26, 2026 3:46:45 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260926 03:46:45.737900 779 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.194.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.194.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.194.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/krb5kdc/rangerkms_127.0.194.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.194.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/krb5kdc/rangerkms_127.0.194.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.194.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.194.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.194.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/krb5kdc/HTTP_127.0.194.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.194.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/krb5kdc/HTTP_127.0.194.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 26 03:46:45 dist-test-slave-n2th krb5kdc[12649](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394405, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260926 03:46:45.923839 779 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260926 03:46:45.946383 779 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260926 03:46:46.801527 779 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260926 03:46:46.801658 779 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/minicluster-data/ranger-kms
I20260926 03:46:46.801717 779 mini_ranger_kms.cc:227] Using postgres at 127.0.194.212:36559
2026-09-26 03:46:47,157 [I] DB FLAVOR :POSTGRES
2026-09-26 03:46:47,157 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 03:46:47,158 [I] Checking connection
2026-09-26 03:46:47,158 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktGRWtf/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.194.212:36559/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-26 03:46:47,416 [I] connection success
2026-09-26 03:46:47,416 [I] --------- Verifying Ranger DB tables ---------
2026-09-26 03:46:47,416 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-26 03:46:47,417 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktGRWtf/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.194.212:36559/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-26 03:46:47,665 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-26 03:46:47,665 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 03:46:47,665 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-26 03:46:47,665 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktGRWtf/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.194.212:36559/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-26 03:46:47,944 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260926 03:46:48.421350 779 mini_ranger_kms.cc:310] Created kms service
I20260926 03:46:48.612399 779 mini_ranger_kms.cc:326] Created kudu user
I20260926 03:46:48.660760 779 mini_ranger_kms.cc:343] Created rangerkms user
Sep 26 03:46:48 dist-test-slave-n2th krb5kdc[12649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394405, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.194.212@KRBTEST.COM
I20260926 03:46:49.067525 779 mini_ranger_kms.cc:384] Added ranger policy
I20260926 03:46:49.067907 779 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/minicluster-data/ranger-kms:/tmp/dist-test-tasktGRWtf/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktGRWtf/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-tasktGRWtf/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260926 03:46:49.067978 779 mini_ranger_kms.cc:240] Using host: 127.0.194.212
I20260926 03:46:49.085225 779 mini_ranger_kms.cc:276] Ranger KMS PID: 13042
I20260926 03:46:49.085364 779 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.194.212:53233
Sep 26, 2026 3:46:50 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 3:46:50 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 3:46:50 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 26, 2026 3:46:50 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 26, 2026 3:46:50 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/minicluster-data/ranger-kms/ews/webapp].
Sep 26, 2026 3:46:50 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260926 03:46:51.345705 779 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.422s user 0.067s sys 0.304s
I20260926 03:46:51.345971 779 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260926 03:46:51.346200 779 mini_ranger_kms.cc:401] 127.0.194.212:53233/kms/v1/keys
Sep 26 03:46:51 dist-test-slave-n2th krb5kdc[12649](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790394411, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.194.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 26 03:46:51 dist-test-slave-n2th krb5kdc[12649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394411, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.194.212@KRBTEST.COM for HTTP/127.0.194.212@KRBTEST.COM
Sep 26 03:46:52 dist-test-slave-n2th krb5kdc[12649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394411, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.194.212@KRBTEST.COM for HTTP/127.0.194.212@KRBTEST.COM
Sep 26 03:46:54 dist-test-slave-n2th krb5kdc[12649](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394414, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.194.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.194.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.194.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/krb5kdc/kudu_127.0.194.254.keytab.
Entry for principal kudu/127.0.194.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/krb5kdc/kudu_127.0.194.254.keytab.
I20260926 03:46:54.519060 779 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
/tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.194.254:40481
--webserver_interface=127.0.194.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/krb5kdc/kudu_127.0.194.254.keytab
--principal=kudu/127.0.194.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.194.212:46639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.194.254:40481
--ranger_config_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.194.212:53233/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:46:54.870471 13093 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:46:54.870888 13093 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:46:54.871047 13093 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:46:54.879079 13093 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:46:54.879200 13093 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:46:54.879300 13093 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:46:54.879391 13093 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:46:54.879480 13093 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:46:54.889802 13093 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.194.212:46639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.194.212:53233/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.194.254:40481
--ranger_config_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/krb5kdc/kudu_127.0.194.254.keytab
--principal=kudu/127.0.194.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.194.254:40481
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.194.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:46:54.891247 13093 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:46:54.892585 13093 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:46:54.904567 13101 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:46:54.904704 13098 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:46:54.904534 13099 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:46:54.905802 13093 server_base.cc:1034] running on GCE node
I20260926 03:46:54.906585 13093 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:46:54.908116 13093 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:46:54.909353 13093 hybrid_clock.cc:648] HybridClock initialized: now 1790394414909320 us; error 51 us; skew 500 ppm
I20260926 03:46:54.909711 13093 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:46:54 dist-test-slave-n2th krb5kdc[12649](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394414, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:46:54.919881 13093 init.cc:377] Logged in from keytab as kudu/127.0.194.254@KRBTEST.COM (short username kudu)
I20260926 03:46:54.921586 13093 webserver.cc:460] Webserver started at http://127.0.194.254:46047/ using document root <none> and password file <none>
I20260926 03:46:54.922304 13093 fs_manager.cc:359] Metadata directory not provided
I20260926 03:46:54.922447 13093 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:46:54.922847 13093 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 03:46:54 dist-test-slave-n2th krb5kdc[12649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394414, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.254@KRBTEST.COM for HTTP/127.0.194.212@KRBTEST.COM
I20260926 03:46:54.988087 13093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/minicluster-data/master-0/data/instance:
uuid: "b27896b679d54f50998598522e611d7c"
format_stamp: "Formatted at 2026-09-26 03:46:54 on dist-test-slave-n2th"
server_key: "8787d45145f36a491af0a59aae1a13b6"
server_key_iv: "33768e94dffc4f91a1f514a3f86a119f"
server_key_version: "kuduclusterkey@0"
I20260926 03:46:54.989012 13093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/minicluster-data/master-0/wal/instance:
uuid: "b27896b679d54f50998598522e611d7c"
format_stamp: "Formatted at 2026-09-26 03:46:54 on dist-test-slave-n2th"
server_key: "8787d45145f36a491af0a59aae1a13b6"
server_key_iv: "33768e94dffc4f91a1f514a3f86a119f"
server_key_version: "kuduclusterkey@0"
I20260926 03:46:54.994558 13093 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260926 03:46:55.015808 13108 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:46:55.017700 13093 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260926 03:46:55.017904 13093 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/minicluster-data/master-0/data,/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/minicluster-data/master-0/wal
uuid: "b27896b679d54f50998598522e611d7c"
format_stamp: "Formatted at 2026-09-26 03:46:54 on dist-test-slave-n2th"
server_key: "8787d45145f36a491af0a59aae1a13b6"
server_key_iv: "33768e94dffc4f91a1f514a3f86a119f"
server_key_version: "kuduclusterkey@0"
I20260926 03:46:55.018184 13093 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:46:55.050215 13093 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:46:55.055843 13093 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:46:55.056214 13093 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:46:55.076833 13093 rpc_server.cc:307] RPC server started. Bound to: 127.0.194.254:40481
I20260926 03:46:55.076834 13159 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.194.254:40481 every 8 connection(s)
I20260926 03:46:55.078624 13093 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/minicluster-data/master-0/data/info.pb
I20260926 03:46:55.079771 779 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu as pid 13093
I20260926 03:46:55.080003 779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/minicluster-data/master-0/wal/instance
I20260926 03:46:55.080816 779 external_mini_cluster.cc:1428] Setting key adadfe7b6fd9406330da8fb08430399c
I20260926 03:46:55.084254 13160 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:46:55.102927 13160 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b27896b679d54f50998598522e611d7c: Bootstrap starting.
I20260926 03:46:55.106679 13160 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b27896b679d54f50998598522e611d7c: Neither blocks nor log segments found. Creating new log.
I20260926 03:46:55.108161 13160 log.cc:824] T 00000000000000000000000000000000 P b27896b679d54f50998598522e611d7c: Log is configured to *not* fsync() on all Append() calls
Sep 26 03:46:55 dist-test-slave-n2th krb5kdc[12649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394414, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.194.254@KRBTEST.COM
I20260926 03:46:55.111547 13160 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b27896b679d54f50998598522e611d7c: No bootstrap required, opened a new log
I20260926 03:46:55.121778 13163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:46:55.088077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46632 (local address 127.0.194.254:40481)
0926 03:46:55.095358 (+ 7281us) server_negotiation.cc:206] Beginning negotiation
0926 03:46:55.095378 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0926 03:46:55.095993 (+ 615us) server_negotiation.cc:407] Connection header received
0926 03:46:55.098462 (+ 2469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:46:55.098498 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:46:55.099129 (+ 631us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:46:55.099686 (+ 557us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:46:55.101563 (+ 1877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:46:55.103735 (+ 2172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:46:55.105743 (+ 2008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:46:55.106919 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:46:55.107120 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:46:55.112084 (+ 4964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:46:55.112098 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:46:55.112120 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:46:55.112167 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:46:55.116342 (+ 4175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:46:55.117387 (+ 1045us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:46:55.117397 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:46:55.117408 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:46:55.117619 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:46:55.118094 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:46:55.118099 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:46:55.118101 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:46:55.118951 (+ 850us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:46:55.119258 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0926 03:46:55.119764 (+ 506us) server_negotiation.cc:299] Negotiation successful
0926 03:46:55.120260 (+ 496us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7000,"thread_start_us":6769,"threads_started":1}
I20260926 03:46:55.122514 13160 raft_consensus.cc:348] T 00000000000000000000000000000000 P b27896b679d54f50998598522e611d7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b27896b679d54f50998598522e611d7c" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 40481 } }
I20260926 03:46:55.123118 13160 raft_consensus.cc:374] T 00000000000000000000000000000000 P b27896b679d54f50998598522e611d7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:46:55.123232 13160 raft_consensus.cc:729] T 00000000000000000000000000000000 P b27896b679d54f50998598522e611d7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b27896b679d54f50998598522e611d7c, State: Initialized, Role: FOLLOWER
I20260926 03:46:55.124019 13160 consensus_queue.cc:260] T 00000000000000000000000000000000 P b27896b679d54f50998598522e611d7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b27896b679d54f50998598522e611d7c" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 40481 } }
I20260926 03:46:55.124291 13160 raft_consensus.cc:388] T 00000000000000000000000000000000 P b27896b679d54f50998598522e611d7c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:46:55.124445 13160 raft_consensus.cc:482] T 00000000000000000000000000000000 P b27896b679d54f50998598522e611d7c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:46:55.124733 13160 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b27896b679d54f50998598522e611d7c [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:46:55.127660 13160 raft_consensus.cc:504] T 00000000000000000000000000000000 P b27896b679d54f50998598522e611d7c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b27896b679d54f50998598522e611d7c" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 40481 } }
I20260926 03:46:55.128198 13160 leader_election.cc:302] T 00000000000000000000000000000000 P b27896b679d54f50998598522e611d7c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b27896b679d54f50998598522e611d7c; no voters:
I20260926 03:46:55.128860 13160 leader_election.cc:288] T 00000000000000000000000000000000 P b27896b679d54f50998598522e611d7c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:46:55.128970 13165 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b27896b679d54f50998598522e611d7c [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:46:55.131618 13160 sys_catalog.cc:564] T 00000000000000000000000000000000 P b27896b679d54f50998598522e611d7c [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:46:55.135044 13165 raft_consensus.cc:686] T 00000000000000000000000000000000 P b27896b679d54f50998598522e611d7c [term 1 LEADER]: Becoming Leader. State: Replica: b27896b679d54f50998598522e611d7c, State: Running, Role: LEADER
I20260926 03:46:55.135962 13165 consensus_queue.cc:237] T 00000000000000000000000000000000 P b27896b679d54f50998598522e611d7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b27896b679d54f50998598522e611d7c" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 40481 } }
I20260926 03:46:55.142278 13167 sys_catalog.cc:455] T 00000000000000000000000000000000 P b27896b679d54f50998598522e611d7c [sys.catalog]: SysCatalogTable state changed. Reason: New leader b27896b679d54f50998598522e611d7c. Latest consensus state: current_term: 1 leader_uuid: "b27896b679d54f50998598522e611d7c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b27896b679d54f50998598522e611d7c" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 40481 } } }
I20260926 03:46:55.143016 13166 sys_catalog.cc:455] T 00000000000000000000000000000000 P b27896b679d54f50998598522e611d7c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b27896b679d54f50998598522e611d7c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b27896b679d54f50998598522e611d7c" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 40481 } } }
I20260926 03:46:55.143394 13166 sys_catalog.cc:458] T 00000000000000000000000000000000 P b27896b679d54f50998598522e611d7c [sys.catalog]: This master's current role is: LEADER
I20260926 03:46:55.143976 13167 sys_catalog.cc:458] T 00000000000000000000000000000000 P b27896b679d54f50998598522e611d7c [sys.catalog]: This master's current role is: LEADER
I20260926 03:46:55.152083 13173 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:46:55.159348 13173 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:46:55.165217 13160 ranger_client.cc:318] Using new properties file: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260926 03:46:55.180330 13173 catalog_manager.cc:1347] Generated new cluster ID: fd8a145d5e384cbfac6d410e3f7edd3e
I20260926 03:46:55.180572 13173 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:46:55.207319 13173 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:46:55.209472 13173 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:46:55.218295 13173 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b27896b679d54f50998598522e611d7c: Generated new TSK 0
I20260926 03:46:55.219377 13173 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.194.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.194.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.194.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/krb5kdc/kudu_127.0.194.193.keytab.
Entry for principal kudu/127.0.194.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/krb5kdc/kudu_127.0.194.193.keytab.
I20260926 03:46:55.999536 779 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
/tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.194.212:53233/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.194.193:0
--local_ip_for_outbound_sockets=127.0.194.193
--webserver_interface=127.0.194.193
--webserver_port=0
--tserver_master_addrs=127.0.194.254:40481
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/krb5kdc/kudu_127.0.194.193.keytab
--principal=kudu/127.0.194.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.194.212:46639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 03:46:56 dist-test-slave-n2th krb5kdc[12649](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790394416, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260926 03:46:56.422600 13205 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:46:56.423058 13205 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:46:56.423239 13205 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:46:56.434950 13205 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:46:56.435179 13205 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:46:56.435418 13205 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.194.193
I20260926 03:46:56.451354 13205 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.194.212:46639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.194.212:53233/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/krb5kdc/kudu_127.0.194.193.keytab
--principal=kudu/127.0.194.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.194.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.194.193
--webserver_port=0
--tserver_master_addrs=127.0.194.254:40481
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.194.193
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:46:56.453197 13205 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:46:56.455014 13205 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:46:56.473812 13216 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:46:56.475301 13218 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:46:56.475708 13215 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:46:56.477646 13205 server_base.cc:1034] running on GCE node
I20260926 03:46:56.478468 13205 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:46:56.480010 13205 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:46:56.481361 13205 hybrid_clock.cc:648] HybridClock initialized: now 1790394416481293 us; error 64 us; skew 500 ppm
I20260926 03:46:56.481786 13205 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:46:56 dist-test-slave-n2th krb5kdc[12649](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394416, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:46:56.486392 13205 init.cc:377] Logged in from keytab as kudu/127.0.194.193@KRBTEST.COM (short username kudu)
I20260926 03:46:56.492276 13205 webserver.cc:460] Webserver started at http://127.0.194.193:45353/ using document root <none> and password file <none>
I20260926 03:46:56.493284 13205 fs_manager.cc:359] Metadata directory not provided
I20260926 03:46:56.493544 13205 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:46:56.494004 13205 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 03:46:56 dist-test-slave-n2th krb5kdc[12649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394416, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.193@KRBTEST.COM for HTTP/127.0.194.212@KRBTEST.COM
I20260926 03:46:56.524724 13205 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/minicluster-data/ts-0/data/instance:
uuid: "5504b5ac53494038b4744fbdf0a92448"
format_stamp: "Formatted at 2026-09-26 03:46:56 on dist-test-slave-n2th"
server_key: "84ef279cc5b67164ef9284376e9b073c"
server_key_iv: "17fd6aa9d31d5fff321814cf066e93ed"
server_key_version: "kuduclusterkey@0"
I20260926 03:46:56.526033 13205 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/minicluster-data/ts-0/wal/instance:
uuid: "5504b5ac53494038b4744fbdf0a92448"
format_stamp: "Formatted at 2026-09-26 03:46:56 on dist-test-slave-n2th"
server_key: "84ef279cc5b67164ef9284376e9b073c"
server_key_iv: "17fd6aa9d31d5fff321814cf066e93ed"
server_key_version: "kuduclusterkey@0"
I20260926 03:46:56.533957 13205 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20260926 03:46:56.567430 13225 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:46:56.569561 13205 fs_manager.cc:725] Time spent opening block manager: real 0.014s user 0.004s sys 0.007s
I20260926 03:46:56.569865 13205 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/minicluster-data/ts-0/wal
uuid: "5504b5ac53494038b4744fbdf0a92448"
format_stamp: "Formatted at 2026-09-26 03:46:56 on dist-test-slave-n2th"
server_key: "84ef279cc5b67164ef9284376e9b073c"
server_key_iv: "17fd6aa9d31d5fff321814cf066e93ed"
server_key_version: "kuduclusterkey@0"
I20260926 03:46:56.570236 13205 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:46:56.596781 13205 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:46:56.601187 13205 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:46:56.601575 13205 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:46:56.602703 13205 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:46:56.604314 13205 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:46:56.604417 13205 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:46:56.604573 13205 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:46:56.604643 13205 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:46:56.687624 13205 rpc_server.cc:307] RPC server started. Bound to: 127.0.194.193:46003
I20260926 03:46:56.689572 13205 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/minicluster-data/ts-0/data/info.pb
I20260926 03:46:56.687635 13337 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.194.193:46003 every 8 connection(s)
I20260926 03:46:56.694458 779 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu as pid 13205
I20260926 03:46:56.694677 779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/minicluster-data/ts-0/wal/instance
I20260926 03:46:56.695787 779 external_mini_cluster.cc:1428] Setting key aec50db6ef9c5b4ec5b8ae1d44b12d16
WARNING: no policy specified for kudu/127.0.194.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.194.194@KRBTEST.COM" created.
Sep 26 03:46:56 dist-test-slave-n2th krb5kdc[12649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394416, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.193@KRBTEST.COM for kudu/127.0.194.254@KRBTEST.COM
I20260926 03:46:56.768524 13341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:46:56.694503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.193:32865 (local address 127.0.194.254:40481)
0926 03:46:56.731189 (+ 36686us) server_negotiation.cc:206] Beginning negotiation
0926 03:46:56.731197 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:46:56.731226 (+ 29us) server_negotiation.cc:407] Connection header received
0926 03:46:56.736033 (+ 4807us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:46:56.736040 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:46:56.736165 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:46:56.736269 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:46:56.742969 (+ 6700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:46:56.744219 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:46:56.748582 (+ 4363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:46:56.749261 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:46:56.749393 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:46:56.756430 (+ 7037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:46:56.756440 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:46:56.756448 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:46:56.756499 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:46:56.763527 (+ 7028us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:46:56.764440 (+ 913us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:46:56.764448 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:46:56.764454 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:46:56.764634 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:46:56.766190 (+ 1556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:46:56.766195 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:46:56.766199 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:46:56.766581 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:46:56.766781 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0926 03:46:56.767878 (+ 1097us) server_negotiation.cc:299] Negotiation successful
0926 03:46:56.768126 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36513,"thread_start_us":37648,"threads_started":1}
I20260926 03:46:56.771235 13340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:46:56.693432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:40481 (local address 127.0.194.193:32865)
0926 03:46:56.727932 (+ 34500us) negotiation.cc:107] Waiting for socket to connect
0926 03:46:56.728003 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0926 03:46:56.735584 (+ 7581us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:46:56.736992 (+ 1408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:46:56.737009 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:46:56.737844 (+ 835us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:46:56.738913 (+ 1069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:46:56.738950 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:46:56.745444 (+ 6494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:46:56.745453 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:46:56.748333 (+ 2880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:46:56.748357 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:46:56.749779 (+ 1422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:46:56.749787 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:46:56.750121 (+ 334us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:46:56.752609 (+ 2488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:46:56.752646 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:46:56.756079 (+ 3433us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:46:56.763772 (+ 7693us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:46:56.763780 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:46:56.763796 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:46:56.764274 (+ 478us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:46:56.765771 (+ 1497us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:46:56.765774 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:46:56.765776 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:46:56.766000 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:46:56.766876 (+ 876us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:46:56.766886 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:46:56.767392 (+ 506us) client_negotiation.cc:769] Sending connection context
0926 03:46:56.769324 (+ 1932us) client_negotiation.cc:240] Negotiation successful
0926 03:46:56.769946 (+ 622us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34049,"thread_start_us":37719,"threads_started":1}
I20260926 03:46:56.773448 13338 heartbeater.cc:347] Connected to a master server at 127.0.194.254:40481
I20260926 03:46:56.773787 13338 heartbeater.cc:464] Registering TS with master...
I20260926 03:46:56.774672 13338 heartbeater.cc:511] Master 127.0.194.254:40481 requested a full tablet report, sending...
I20260926 03:46:56.777652 13125 ts_manager.cc:194] Registered new tserver with Master: 5504b5ac53494038b4744fbdf0a92448 (127.0.194.193:46003)
I20260926 03:46:56.779886 13125 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.194.193@KRBTEST.COM'} at 127.0.194.193:32865
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.194.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/krb5kdc/kudu_127.0.194.194.keytab.
Entry for principal kudu/127.0.194.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/krb5kdc/kudu_127.0.194.194.keytab.
I20260926 03:46:56.798112 779 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
/tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.194.212:53233/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.194.194:0
--local_ip_for_outbound_sockets=127.0.194.194
--webserver_interface=127.0.194.194
--webserver_port=0
--tserver_master_addrs=127.0.194.254:40481
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/krb5kdc/kudu_127.0.194.194.keytab
--principal=kudu/127.0.194.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.194.212:46639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 03:46:56 dist-test-slave-n2th krb5kdc[12649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394416, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.254@KRBTEST.COM for HTTP/127.0.194.212@KRBTEST.COM
Sep 26 03:46:57 dist-test-slave-n2th krb5kdc[12649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394416, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.254@KRBTEST.COM for HTTP/127.0.194.212@KRBTEST.COM
W20260926 03:46:57.296801 13344 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:46:57.297333 13344 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:46:57.297530 13344 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:46:57.309315 13344 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:46:57.309556 13344 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:46:57.309798 13344 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.194.194
I20260926 03:46:57.325824 13344 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.194.212:46639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.194.212:53233/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/krb5kdc/kudu_127.0.194.194.keytab
--principal=kudu/127.0.194.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.194.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.194.194
--webserver_port=0
--tserver_master_addrs=127.0.194.254:40481
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.194.194
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:46:57.327919 13344 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:46:57.329826 13344 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:46:57.366168 13351 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:46:57.367813 13352 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:46:57.383282 13355 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:46:57.383656 13344 server_base.cc:1034] running on GCE node
I20260926 03:46:57.384693 13344 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:46:57.391155 13344 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:46:57.392541 13344 hybrid_clock.cc:648] HybridClock initialized: now 1790394417392436 us; error 110 us; skew 500 ppm
I20260926 03:46:57.393016 13344 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:46:57 dist-test-slave-n2th krb5kdc[12649](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394417, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:46:57.397426 13344 init.cc:377] Logged in from keytab as kudu/127.0.194.194@KRBTEST.COM (short username kudu)
I20260926 03:46:57.411875 13344 webserver.cc:460] Webserver started at http://127.0.194.194:33883/ using document root <none> and password file <none>
I20260926 03:46:57.412914 13344 fs_manager.cc:359] Metadata directory not provided
I20260926 03:46:57.413065 13344 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:46:57.413507 13344 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 03:46:57 dist-test-slave-n2th krb5kdc[12649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394417, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.194@KRBTEST.COM for HTTP/127.0.194.212@KRBTEST.COM
I20260926 03:46:57.437130 13344 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/minicluster-data/ts-1/data/instance:
uuid: "2e24b8bd66f647e98e3147831ef50fbe"
format_stamp: "Formatted at 2026-09-26 03:46:57 on dist-test-slave-n2th"
server_key: "fb7fd2956d758ae515cf9f8e76b1fb9c"
server_key_iv: "6a895db182b5a94a17ca7197df9edf65"
server_key_version: "kuduclusterkey@0"
I20260926 03:46:57.438405 13344 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/minicluster-data/ts-1/wal/instance:
uuid: "2e24b8bd66f647e98e3147831ef50fbe"
format_stamp: "Formatted at 2026-09-26 03:46:57 on dist-test-slave-n2th"
server_key: "fb7fd2956d758ae515cf9f8e76b1fb9c"
server_key_iv: "6a895db182b5a94a17ca7197df9edf65"
server_key_version: "kuduclusterkey@0"
I20260926 03:46:57.445803 13344 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.003s sys 0.002s
I20260926 03:46:57.477898 13364 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:46:57.480145 13344 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.003s sys 0.002s
I20260926 03:46:57.480463 13344 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/minicluster-data/ts-1/wal
uuid: "2e24b8bd66f647e98e3147831ef50fbe"
format_stamp: "Formatted at 2026-09-26 03:46:57 on dist-test-slave-n2th"
server_key: "fb7fd2956d758ae515cf9f8e76b1fb9c"
server_key_iv: "6a895db182b5a94a17ca7197df9edf65"
server_key_version: "kuduclusterkey@0"
I20260926 03:46:57.480829 13344 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:46:57.507140 13344 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:46:57.511487 13344 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:46:57.511811 13344 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:46:57.512813 13344 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:46:57.514223 13344 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:46:57.514297 13344 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:46:57.514454 13344 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:46:57.514503 13344 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:46:57.548822 13344 rpc_server.cc:307] RPC server started. Bound to: 127.0.194.194:35349
I20260926 03:46:57.549027 13480 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.194.194:35349 every 8 connection(s)
I20260926 03:46:57.550163 13344 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/minicluster-data/ts-1/data/info.pb
I20260926 03:46:57.560217 779 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu as pid 13344
I20260926 03:46:57.560400 779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/minicluster-data/ts-1/wal/instance
I20260926 03:46:57.560979 779 external_mini_cluster.cc:1428] Setting key d155f8bf475fa0cf3fe5b5a45c9bd1b6
Sep 26 03:46:57 dist-test-slave-n2th krb5kdc[12649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394417, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.194@KRBTEST.COM for kudu/127.0.194.254@KRBTEST.COM
I20260926 03:46:57.572467 13483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:46:57.553064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.194:39689 (local address 127.0.194.254:40481)
0926 03:46:57.553543 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0926 03:46:57.553548 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:46:57.554464 (+ 916us) server_negotiation.cc:407] Connection header received
0926 03:46:57.556365 (+ 1901us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:46:57.556371 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:46:57.556477 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:46:57.556597 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:46:57.558137 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:46:57.558974 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:46:57.561127 (+ 2153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:46:57.561562 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:46:57.561667 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:46:57.565219 (+ 3552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:46:57.565228 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:46:57.565236 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:46:57.565281 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:46:57.569142 (+ 3861us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:46:57.569932 (+ 790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:46:57.569942 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:46:57.569948 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:46:57.570107 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:46:57.570522 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:46:57.570530 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:46:57.570536 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:46:57.570969 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:46:57.571138 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0926 03:46:57.572038 (+ 900us) server_negotiation.cc:299] Negotiation successful
0926 03:46:57.572208 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":383,"thread_start_us":265,"threads_started":1}
I20260926 03:46:57.574046 13484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:46:57.553529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:40481 (local address 127.0.194.194:39689)
0926 03:46:57.554281 (+ 752us) negotiation.cc:107] Waiting for socket to connect
0926 03:46:57.554334 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 03:46:57.556085 (+ 1751us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:46:57.556839 (+ 754us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:46:57.556847 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:46:57.557387 (+ 540us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:46:57.557969 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:46:57.557987 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:46:57.559124 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:46:57.559130 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:46:57.560941 (+ 1811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:46:57.560990 (+ 49us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:46:57.561816 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:46:57.561823 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:46:57.562071 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:46:57.562838 (+ 767us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:46:57.562869 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:46:57.564991 (+ 2122us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:46:57.569292 (+ 4301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:46:57.569299 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:46:57.569313 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:46:57.569804 (+ 491us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:46:57.570218 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:46:57.570222 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:46:57.570224 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:46:57.570416 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:46:57.571098 (+ 682us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:46:57.571104 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:46:57.571616 (+ 512us) client_negotiation.cc:769] Sending connection context
0926 03:46:57.571966 (+ 350us) client_negotiation.cc:240] Negotiation successful
0926 03:46:57.572576 (+ 610us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":320,"threads_started":1}
I20260926 03:46:57.575621 13481 heartbeater.cc:347] Connected to a master server at 127.0.194.254:40481
I20260926 03:46:57.575965 13481 heartbeater.cc:464] Registering TS with master...
I20260926 03:46:57.576818 13481 heartbeater.cc:511] Master 127.0.194.254:40481 requested a full tablet report, sending...
I20260926 03:46:57.578404 13125 ts_manager.cc:194] Registered new tserver with Master: 2e24b8bd66f647e98e3147831ef50fbe (127.0.194.194:35349)
I20260926 03:46:57.579263 13125 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.194.194@KRBTEST.COM'} at 127.0.194.194:39689
WARNING: no policy specified for kudu/127.0.194.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.194.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.194.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/krb5kdc/kudu_127.0.194.195.keytab.
Entry for principal kudu/127.0.194.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/krb5kdc/kudu_127.0.194.195.keytab.
I20260926 03:46:57.614025 779 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
/tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.194.212:53233/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.194.195:0
--local_ip_for_outbound_sockets=127.0.194.195
--webserver_interface=127.0.194.195
--webserver_port=0
--tserver_master_addrs=127.0.194.254:40481
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/krb5kdc/kudu_127.0.194.195.keytab
--principal=kudu/127.0.194.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.194.212:46639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 03:46:57.784210 13338 heartbeater.cc:503] Master 127.0.194.254:40481 was elected leader, sending a full tablet report...
W20260926 03:46:57.949496 13487 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:46:57.949802 13487 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:46:57.949896 13487 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:46:57.957348 13487 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:46:57.957469 13487 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:46:57.957583 13487 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.194.195
I20260926 03:46:57.968261 13487 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.194.212:46639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.194.212:53233/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/krb5kdc/kudu_127.0.194.195.keytab
--principal=kudu/127.0.194.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.194.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.194.195
--webserver_port=0
--tserver_master_addrs=127.0.194.254:40481
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.194.195
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:46:57.969571 13487 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:46:57.970889 13487 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:46:57.984122 13493 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:46:57.984125 13495 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:46:57.984169 13492 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:46:57.985193 13487 server_base.cc:1034] running on GCE node
I20260926 03:46:57.985903 13487 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:46:57.987071 13487 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:46:57.988296 13487 hybrid_clock.cc:648] HybridClock initialized: now 1790394417988252 us; error 42 us; skew 500 ppm
I20260926 03:46:57.988646 13487 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:46:57 dist-test-slave-n2th krb5kdc[12649](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394417, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:46:57.991855 13487 init.cc:377] Logged in from keytab as kudu/127.0.194.195@KRBTEST.COM (short username kudu)
I20260926 03:46:57.993342 13487 webserver.cc:460] Webserver started at http://127.0.194.195:46371/ using document root <none> and password file <none>
I20260926 03:46:57.994029 13487 fs_manager.cc:359] Metadata directory not provided
I20260926 03:46:57.994134 13487 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:46:57.994400 13487 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 03:46:57 dist-test-slave-n2th krb5kdc[12649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394417, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.195@KRBTEST.COM for HTTP/127.0.194.212@KRBTEST.COM
I20260926 03:46:58.010656 13487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/minicluster-data/ts-2/data/instance:
uuid: "0f450de5b0ae40d4bb4e3e36ceddbd07"
format_stamp: "Formatted at 2026-09-26 03:46:58 on dist-test-slave-n2th"
server_key: "f0dbb789cf617fc98247f41fdc06f757"
server_key_iv: "dfa67067df09f6963f2dd93fc2e24bc6"
server_key_version: "kuduclusterkey@0"
I20260926 03:46:58.011601 13487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/minicluster-data/ts-2/wal/instance:
uuid: "0f450de5b0ae40d4bb4e3e36ceddbd07"
format_stamp: "Formatted at 2026-09-26 03:46:58 on dist-test-slave-n2th"
server_key: "f0dbb789cf617fc98247f41fdc06f757"
server_key_iv: "dfa67067df09f6963f2dd93fc2e24bc6"
server_key_version: "kuduclusterkey@0"
I20260926 03:46:58.017230 13487 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260926 03:46:58.033633 13502 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:46:58.035374 13487 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260926 03:46:58.035576 13487 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/minicluster-data/ts-2/wal
uuid: "0f450de5b0ae40d4bb4e3e36ceddbd07"
format_stamp: "Formatted at 2026-09-26 03:46:58 on dist-test-slave-n2th"
server_key: "f0dbb789cf617fc98247f41fdc06f757"
server_key_iv: "dfa67067df09f6963f2dd93fc2e24bc6"
server_key_version: "kuduclusterkey@0"
I20260926 03:46:58.035804 13487 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:46:58.056308 13487 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:46:58.060683 13487 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:46:58.061017 13487 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:46:58.061995 13487 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:46:58.063401 13487 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:46:58.063509 13487 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:46:58.063655 13487 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:46:58.063727 13487 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:46:58.100978 13487 rpc_server.cc:307] RPC server started. Bound to: 127.0.194.195:40201
I20260926 03:46:58.101030 13614 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.194.195:40201 every 8 connection(s)
I20260926 03:46:58.102363 13487 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/minicluster-data/ts-2/data/info.pb
I20260926 03:46:58.108747 779 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu as pid 13487
I20260926 03:46:58.108909 779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/minicluster-data/ts-2/wal/instance
I20260926 03:46:58.109577 779 external_mini_cluster.cc:1428] Setting key daf19da3e54b55e3a86dde35f62cdd7d
Sep 26 03:46:58 dist-test-slave-n2th krb5kdc[12649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394417, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.195@KRBTEST.COM for kudu/127.0.194.254@KRBTEST.COM
I20260926 03:46:58.124168 13617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:46:58.105082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.195:39545 (local address 127.0.194.254:40481)
0926 03:46:58.105585 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0926 03:46:58.105591 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:46:58.106661 (+ 1070us) server_negotiation.cc:407] Connection header received
0926 03:46:58.108687 (+ 2026us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:46:58.108692 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:46:58.108802 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:46:58.108966 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:46:58.111009 (+ 2043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:46:58.111965 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:46:58.114166 (+ 2201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:46:58.114620 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:46:58.114723 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:46:58.118179 (+ 3456us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:46:58.118184 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:46:58.118188 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:46:58.118229 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:46:58.120882 (+ 2653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:46:58.121687 (+ 805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:46:58.121691 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:46:58.121694 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:46:58.121828 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:46:58.122250 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:46:58.122253 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:46:58.122255 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:46:58.122532 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:46:58.122646 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0926 03:46:58.123512 (+ 866us) server_negotiation.cc:299] Negotiation successful
0926 03:46:58.123772 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":282,"threads_started":1}
I20260926 03:46:58.125365 13618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:46:58.105655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:40481 (local address 127.0.194.195:39545)
0926 03:46:58.106449 (+ 794us) negotiation.cc:107] Waiting for socket to connect
0926 03:46:58.106488 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 03:46:58.108279 (+ 1791us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:46:58.109323 (+ 1044us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:46:58.109337 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:46:58.109978 (+ 641us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:46:58.110736 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:46:58.110789 (+ 53us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:46:58.112124 (+ 1335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:46:58.112129 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:46:58.114021 (+ 1892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:46:58.114035 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:46:58.114905 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:46:58.114911 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:46:58.115130 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:46:58.115797 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:46:58.115825 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:46:58.117928 (+ 2103us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:46:58.121068 (+ 3140us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:46:58.121076 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:46:58.121092 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:46:58.121557 (+ 465us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:46:58.121934 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:46:58.121937 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:46:58.121939 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:46:58.122146 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:46:58.122639 (+ 493us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:46:58.122646 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:46:58.123064 (+ 418us) client_negotiation.cc:769] Sending connection context
0926 03:46:58.123392 (+ 328us) client_negotiation.cc:240] Negotiation successful
0926 03:46:58.123875 (+ 483us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":540,"thread_start_us":296,"threads_started":1}
I20260926 03:46:58.126971 13615 heartbeater.cc:347] Connected to a master server at 127.0.194.254:40481
I20260926 03:46:58.127305 13615 heartbeater.cc:464] Registering TS with master...
I20260926 03:46:58.128062 13615 heartbeater.cc:511] Master 127.0.194.254:40481 requested a full tablet report, sending...
I20260926 03:46:58.129539 13125 ts_manager.cc:194] Registered new tserver with Master: 0f450de5b0ae40d4bb4e3e36ceddbd07 (127.0.194.195:40201)
I20260926 03:46:58.130374 13125 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.194.195@KRBTEST.COM'} at 127.0.194.195:39545
I20260926 03:46:58.138896 779 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 03:46:58.140733 779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu pid 13205
Sep 26 03:46:58 dist-test-slave-n2th krb5kdc[12649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394414, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.194.193@KRBTEST.COM
I20260926 03:46:58.158617 13620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:46:58.141754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52870 (local address 127.0.194.193:46003)
0926 03:46:58.142379 (+ 625us) server_negotiation.cc:206] Beginning negotiation
0926 03:46:58.142387 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:46:58.142413 (+ 26us) server_negotiation.cc:407] Connection header received
0926 03:46:58.142476 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:46:58.142483 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:46:58.142693 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:46:58.142917 (+ 224us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:46:58.144436 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:46:58.145906 (+ 1470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:46:58.147501 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:46:58.148115 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:46:58.148268 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:46:58.152640 (+ 4372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:46:58.152653 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:46:58.152677 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:46:58.152718 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:46:58.155519 (+ 2801us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:46:58.156317 (+ 798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:46:58.156325 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:46:58.156327 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:46:58.156485 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:46:58.156905 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:46:58.156912 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:46:58.156916 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:46:58.157278 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:46:58.157616 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0926 03:46:58.157918 (+ 302us) server_negotiation.cc:299] Negotiation successful
0926 03:46:58.158186 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":473,"thread_start_us":334,"threads_started":1}
I20260926 03:46:58.159967 13333 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:46:58.243947 779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu with pid 13205
I20260926 03:46:58.259899 779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu pid 13344
Sep 26 03:46:58 dist-test-slave-n2th krb5kdc[12649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394414, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.194.194@KRBTEST.COM
I20260926 03:46:58.277156 13622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:46:58.260793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46412 (local address 127.0.194.194:35349)
0926 03:46:58.261311 (+ 518us) server_negotiation.cc:206] Beginning negotiation
0926 03:46:58.261318 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:46:58.261342 (+ 24us) server_negotiation.cc:407] Connection header received
0926 03:46:58.261398 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:46:58.261404 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:46:58.261641 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:46:58.261811 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:46:58.263205 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:46:58.264688 (+ 1483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:46:58.266251 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:46:58.267005 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:46:58.267117 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:46:58.271685 (+ 4568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:46:58.271700 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:46:58.271715 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:46:58.271756 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:46:58.274450 (+ 2694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:46:58.275258 (+ 808us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:46:58.275266 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:46:58.275268 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:46:58.275418 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:46:58.275743 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:46:58.275746 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:46:58.275748 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:46:58.276101 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:46:58.276340 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0926 03:46:58.276551 (+ 211us) server_negotiation.cc:299] Negotiation successful
0926 03:46:58.276752 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":236,"threads_started":1}
I20260926 03:46:58.278458 13476 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:46:58.366461 779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu with pid 13344
I20260926 03:46:58.382035 779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu pid 13487
Sep 26 03:46:58 dist-test-slave-n2th krb5kdc[12649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394414, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.194.195@KRBTEST.COM
I20260926 03:46:58.400183 13624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:46:58.383128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34440 (local address 127.0.194.195:40201)
0926 03:46:58.383817 (+ 689us) server_negotiation.cc:206] Beginning negotiation
0926 03:46:58.383824 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:46:58.383854 (+ 30us) server_negotiation.cc:407] Connection header received
0926 03:46:58.383935 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:46:58.383941 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:46:58.384161 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:46:58.384288 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:46:58.385635 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:46:58.386649 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:46:58.388065 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:46:58.388636 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:46:58.388735 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:46:58.393405 (+ 4670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:46:58.393420 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:46:58.393437 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:46:58.393478 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:46:58.397238 (+ 3760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:46:58.398224 (+ 986us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:46:58.398232 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:46:58.398234 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:46:58.398380 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:46:58.398727 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:46:58.398776 (+ 49us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:46:58.398779 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:46:58.399145 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:46:58.399384 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0926 03:46:58.399672 (+ 288us) server_negotiation.cc:299] Negotiation successful
0926 03:46:58.399893 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":545,"thread_start_us":389,"threads_started":1}
I20260926 03:46:58.401495 13610 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:46:58.485172 779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu with pid 13487
I20260926 03:46:58.500774 779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu pid 13093
I20260926 03:46:58.502174 13155 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:46:58.608649 779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu with pid 13093
I20260926 03:46:58.630060 779 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
/tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.194.254:40481
--webserver_interface=127.0.194.254
--webserver_port=46047
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/krb5kdc/kudu_127.0.194.254.keytab
--principal=kudu/127.0.194.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.194.212:46639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.194.254:40481
--ranger_config_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.194.212:53233/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.194.212:53233/kms with env {KRB5CCNAME=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:46:58.962364 13627 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:46:58.962745 13627 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:46:58.962956 13627 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:46:58.970386 13627 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:46:58.970556 13627 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:46:58.970674 13627 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:46:58.970798 13627 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:46:58.970890 13627 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:46:58.981323 13627 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.194.212:46639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.194.212:53233/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.194.254:40481
--ranger_config_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/krb5kdc/kudu_127.0.194.254.keytab
--principal=kudu/127.0.194.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.194.254:40481
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.194.254
--webserver_port=46047
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:46:58.982781 13627 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:46:58.984189 13627 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:46:58.996555 13632 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:46:58.996690 13633 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:46:58.996661 13635 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:46:58.997876 13627 server_base.cc:1034] running on GCE node
I20260926 03:46:58.998682 13627 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:46:59.000267 13627 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:46:59.001484 13627 hybrid_clock.cc:648] HybridClock initialized: now 1790394419001445 us; error 43 us; skew 500 ppm
I20260926 03:46:59.001922 13627 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:46:59 dist-test-slave-n2th krb5kdc[12649](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394419, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:46:59.005261 13627 init.cc:377] Logged in from keytab as kudu/127.0.194.254@KRBTEST.COM (short username kudu)
I20260926 03:46:59.006973 13627 webserver.cc:460] Webserver started at http://127.0.194.254:46047/ using document root <none> and password file <none>
I20260926 03:46:59.007627 13627 fs_manager.cc:359] Metadata directory not provided
I20260926 03:46:59.007751 13627 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 03:46:59 dist-test-slave-n2th krb5kdc[12649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394419, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.254@KRBTEST.COM for HTTP/127.0.194.212@KRBTEST.COM
I20260926 03:46:59.078666 13627 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 03:46:59.081911 13643 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:46:59.083624 13627 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 03:46:59.083842 13627 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/minicluster-data/master-0/data,/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/minicluster-data/master-0/wal
uuid: "b27896b679d54f50998598522e611d7c"
format_stamp: "Formatted at 2026-09-26 03:46:54 on dist-test-slave-n2th"
server_key: "8787d45145f36a491af0a59aae1a13b6"
server_key_iv: "33768e94dffc4f91a1f514a3f86a119f"
server_key_version: "kuduclusterkey@0"
I20260926 03:46:59.084918 13627 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:46:59.104506 13627 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:46:59.108582 13627 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:46:59.108897 13627 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:46:59.128585 13627 rpc_server.cc:307] RPC server started. Bound to: 127.0.194.254:40481
I20260926 03:46:59.128635 13694 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.194.254:40481 every 8 connection(s)
I20260926 03:46:59.130548 13627 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/minicluster-data/master-0/data/info.pb
I20260926 03:46:59.134538 779 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu as pid 13627
I20260926 03:46:59.135643 779 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
/tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.194.212:53233/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.194.193:46003
--local_ip_for_outbound_sockets=127.0.194.193
--tserver_master_addrs=127.0.194.254:40481
--webserver_port=45353
--webserver_interface=127.0.194.193
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/krb5kdc/kudu_127.0.194.193.keytab
--principal=kudu/127.0.194.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.194.212:46639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.194.212:53233/kms with env {KRB5CCNAME=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 03:46:59.140321 13695 sys_catalog.cc:263] Verifying existing consensus state
I20260926 03:46:59.145465 13695 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b27896b679d54f50998598522e611d7c: Bootstrap starting.
I20260926 03:46:59.165818 13695 log.cc:824] T 00000000000000000000000000000000 P b27896b679d54f50998598522e611d7c: Log is configured to *not* fsync() on all Append() calls
I20260926 03:46:59.175071 13695 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b27896b679d54f50998598522e611d7c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 03:46:59.175653 13695 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b27896b679d54f50998598522e611d7c: Bootstrap complete.
I20260926 03:46:59.184913 13695 raft_consensus.cc:348] T 00000000000000000000000000000000 P b27896b679d54f50998598522e611d7c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b27896b679d54f50998598522e611d7c" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 40481 } }
I20260926 03:46:59.185787 13695 raft_consensus.cc:729] T 00000000000000000000000000000000 P b27896b679d54f50998598522e611d7c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b27896b679d54f50998598522e611d7c, State: Initialized, Role: FOLLOWER
I20260926 03:46:59.186456 13695 consensus_queue.cc:260] T 00000000000000000000000000000000 P b27896b679d54f50998598522e611d7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b27896b679d54f50998598522e611d7c" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 40481 } }
I20260926 03:46:59.186717 13695 raft_consensus.cc:388] T 00000000000000000000000000000000 P b27896b679d54f50998598522e611d7c [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:46:59.186861 13695 raft_consensus.cc:482] T 00000000000000000000000000000000 P b27896b679d54f50998598522e611d7c [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:46:59.187075 13695 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b27896b679d54f50998598522e611d7c [term 1 FOLLOWER]: Advancing to term 2
I20260926 03:46:59.191758 13695 raft_consensus.cc:504] T 00000000000000000000000000000000 P b27896b679d54f50998598522e611d7c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b27896b679d54f50998598522e611d7c" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 40481 } }
I20260926 03:46:59.192322 13695 leader_election.cc:302] T 00000000000000000000000000000000 P b27896b679d54f50998598522e611d7c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b27896b679d54f50998598522e611d7c; no voters:
I20260926 03:46:59.192921 13695 leader_election.cc:288] T 00000000000000000000000000000000 P b27896b679d54f50998598522e611d7c [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 03:46:59.193122 13699 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b27896b679d54f50998598522e611d7c [term 2 FOLLOWER]: Leader election won for term 2
I20260926 03:46:59.194242 13699 raft_consensus.cc:686] T 00000000000000000000000000000000 P b27896b679d54f50998598522e611d7c [term 2 LEADER]: Becoming Leader. State: Replica: b27896b679d54f50998598522e611d7c, State: Running, Role: LEADER
I20260926 03:46:59.194926 13699 consensus_queue.cc:237] T 00000000000000000000000000000000 P b27896b679d54f50998598522e611d7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b27896b679d54f50998598522e611d7c" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 40481 } }
I20260926 03:46:59.195082 13695 sys_catalog.cc:564] T 00000000000000000000000000000000 P b27896b679d54f50998598522e611d7c [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:46:59.197821 13700 sys_catalog.cc:455] T 00000000000000000000000000000000 P b27896b679d54f50998598522e611d7c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "b27896b679d54f50998598522e611d7c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b27896b679d54f50998598522e611d7c" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 40481 } } }
I20260926 03:46:59.197821 13701 sys_catalog.cc:455] T 00000000000000000000000000000000 P b27896b679d54f50998598522e611d7c [sys.catalog]: SysCatalogTable state changed. Reason: New leader b27896b679d54f50998598522e611d7c. Latest consensus state: current_term: 2 leader_uuid: "b27896b679d54f50998598522e611d7c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b27896b679d54f50998598522e611d7c" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 40481 } } }
I20260926 03:46:59.198279 13700 sys_catalog.cc:458] T 00000000000000000000000000000000 P b27896b679d54f50998598522e611d7c [sys.catalog]: This master's current role is: LEADER
I20260926 03:46:59.198343 13701 sys_catalog.cc:458] T 00000000000000000000000000000000 P b27896b679d54f50998598522e611d7c [sys.catalog]: This master's current role is: LEADER
I20260926 03:46:59.203843 13705 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:46:59.215191 13705 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:46:59.216689 13705 catalog_manager.cc:1259] Loaded cluster ID: fd8a145d5e384cbfac6d410e3f7edd3e
I20260926 03:46:59.216879 13705 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:46:59.221024 13705 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:46:59.222918 13705 catalog_manager.cc:5879] T 00000000000000000000000000000000 P b27896b679d54f50998598522e611d7c: Loaded TSK: 0
I20260926 03:46:59.224783 13705 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 03:46:59.229878 13695 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
W20260926 03:46:59.568329 13697 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:46:59.568935 13697 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:46:59.569161 13697 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:46:59.582921 13697 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:46:59.583180 13697 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:46:59.583453 13697 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.194.193
I20260926 03:46:59.602026 13697 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.194.212:46639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.194.212:53233/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/krb5kdc/kudu_127.0.194.193.keytab
--principal=kudu/127.0.194.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.194.193:46003
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.194.193
--webserver_port=45353
--tserver_master_addrs=127.0.194.254:40481
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.194.193
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:46:59.603945 13697 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:46:59.605909 13697 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:46:59.637140 13732 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:46:59.639956 13733 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:46:59.649384 13697 server_base.cc:1034] running on GCE node
W20260926 03:46:59.649933 13735 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:46:59.650888 13697 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:46:59.652647 13697 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:46:59.656068 13697 hybrid_clock.cc:648] HybridClock initialized: now 1790394419655995 us; error 61 us; skew 500 ppm
I20260926 03:46:59.656562 13697 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:46:59 dist-test-slave-n2th krb5kdc[12649](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394419, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:46:59.661149 13697 init.cc:377] Logged in from keytab as kudu/127.0.194.193@KRBTEST.COM (short username kudu)
I20260926 03:46:59.667529 13697 webserver.cc:460] Webserver started at http://127.0.194.193:45353/ using document root <none> and password file <none>
I20260926 03:46:59.668465 13697 fs_manager.cc:359] Metadata directory not provided
I20260926 03:46:59.668655 13697 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 03:46:59 dist-test-slave-n2th krb5kdc[12649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394419, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.193@KRBTEST.COM for HTTP/127.0.194.212@KRBTEST.COM
I20260926 03:46:59.705586 13697 fs_manager.cc:709] Time spent opening directory manager: real 0.013s user 0.004s sys 0.006s
I20260926 03:46:59.727871 13743 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:46:59.729962 13697 fs_manager.cc:725] Time spent opening block manager: real 0.022s user 0.008s sys 0.000s
I20260926 03:46:59.730237 13697 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/minicluster-data/ts-0/wal
uuid: "5504b5ac53494038b4744fbdf0a92448"
format_stamp: "Formatted at 2026-09-26 03:46:56 on dist-test-slave-n2th"
server_key: "84ef279cc5b67164ef9284376e9b073c"
server_key_iv: "17fd6aa9d31d5fff321814cf066e93ed"
server_key_version: "kuduclusterkey@0"
I20260926 03:46:59.731837 13697 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:46:59.774868 13697 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:46:59.780540 13697 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:46:59.780972 13697 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:46:59.798676 13697 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:46:59.808209 13697 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:46:59.808444 13697 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260926 03:46:59.808657 13697 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:46:59.808820 13697 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:46:59.876482 13697 rpc_server.cc:307] RPC server started. Bound to: 127.0.194.193:46003
I20260926 03:46:59.876513 13856 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.194.193:46003 every 8 connection(s)
I20260926 03:46:59.878513 13697 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/minicluster-data/ts-0/data/info.pb
I20260926 03:46:59.879582 779 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu as pid 13697
I20260926 03:46:59.880391 779 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
/tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.194.212:53233/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.194.194:35349
--local_ip_for_outbound_sockets=127.0.194.194
--tserver_master_addrs=127.0.194.254:40481
--webserver_port=33883
--webserver_interface=127.0.194.194
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/krb5kdc/kudu_127.0.194.194.keytab
--principal=kudu/127.0.194.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.194.212:46639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.194.212:53233/kms with env {KRB5CCNAME=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 03:46:59 dist-test-slave-n2th krb5kdc[12649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394419, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.193@KRBTEST.COM for kudu/127.0.194.254@KRBTEST.COM
I20260926 03:46:59.938781 13858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:46:59.888446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:40481 (local address 127.0.194.193:42309)
0926 03:46:59.892933 (+ 4487us) negotiation.cc:107] Waiting for socket to connect
0926 03:46:59.892989 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0926 03:46:59.896498 (+ 3509us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:46:59.904356 (+ 7858us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:46:59.904371 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:46:59.905117 (+ 746us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:46:59.907460 (+ 2343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:46:59.907492 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:46:59.916628 (+ 9136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:46:59.916635 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:46:59.918854 (+ 2219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:46:59.918877 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:46:59.920333 (+ 1456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:46:59.920342 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:46:59.920646 (+ 304us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:46:59.926989 (+ 6343us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:46:59.927025 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:46:59.929584 (+ 2559us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:46:59.933449 (+ 3865us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:46:59.933459 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:46:59.933477 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:46:59.934113 (+ 636us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:46:59.934698 (+ 585us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:46:59.934703 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:46:59.934707 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:46:59.935022 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:46:59.935874 (+ 852us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:46:59.935882 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:46:59.936340 (+ 458us) client_negotiation.cc:769] Sending connection context
0926 03:46:59.936699 (+ 359us) client_negotiation.cc:240] Negotiation successful
0926 03:46:59.937309 (+ 610us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4149,"thread_start_us":3895,"threads_started":1}
I20260926 03:46:59.938774 13859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:46:59.889756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.193:42309 (local address 127.0.194.254:40481)
0926 03:46:59.895299 (+ 5543us) server_negotiation.cc:206] Beginning negotiation
0926 03:46:59.895319 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0926 03:46:59.895378 (+ 59us) server_negotiation.cc:407] Connection header received
0926 03:46:59.897400 (+ 2022us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:46:59.897421 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:46:59.897962 (+ 541us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:46:59.898394 (+ 432us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:46:59.908412 (+ 10018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:46:59.916277 (+ 7865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:46:59.919108 (+ 2831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:46:59.920033 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:46:59.920282 (+ 249us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:46:59.929937 (+ 9655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:46:59.929946 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:46:59.929965 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:46:59.930015 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:46:59.933252 (+ 3237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:46:59.934320 (+ 1068us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:46:59.934331 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:46:59.934339 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:46:59.934547 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:46:59.935171 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:46:59.935176 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:46:59.935179 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:46:59.935709 (+ 530us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:46:59.935945 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0926 03:46:59.936933 (+ 988us) server_negotiation.cc:299] Negotiation successful
0926 03:46:59.937309 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5212,"thread_start_us":4972,"threads_started":1}
I20260926 03:46:59.942329 13857 heartbeater.cc:347] Connected to a master server at 127.0.194.254:40481
I20260926 03:46:59.942651 13857 heartbeater.cc:464] Registering TS with master...
I20260926 03:46:59.943428 13857 heartbeater.cc:511] Master 127.0.194.254:40481 requested a full tablet report, sending...
W20260926 03:46:59.946135 13857 heartbeater.cc:644] Failed to heartbeat to 127.0.194.254:40481 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260926 03:46:59.946380 13857 heartbeater.cc:464] Registering TS with master...
I20260926 03:46:59.946672 13857 heartbeater.cc:511] Master 127.0.194.254:40481 requested a full tablet report, sending...
I20260926 03:46:59.947818 13857 heartbeater.cc:464] Registering TS with master...
I20260926 03:46:59.948081 13857 heartbeater.cc:511] Master 127.0.194.254:40481 requested a full tablet report, sending...
W20260926 03:46:59.949097 13857 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 03:47:00.315120 13860 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:47:00.315439 13860 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:47:00.315599 13860 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:47:00.323539 13860 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:47:00.323732 13860 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:47:00.323958 13860 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.194.194
I20260926 03:47:00.337399 13860 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.194.212:46639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.194.212:53233/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/krb5kdc/kudu_127.0.194.194.keytab
--principal=kudu/127.0.194.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.194.194:35349
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.194.194
--webserver_port=33883
--tserver_master_addrs=127.0.194.254:40481
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.194.194
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:47:00.339004 13860 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:47:00.340520 13860 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:47:00.359583 13875 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:47:00.364360 13876 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:47:00.367286 13878 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:47:00.367936 13860 server_base.cc:1034] running on GCE node
I20260926 03:47:00.368841 13860 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:47:00.370260 13860 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:47:00.371532 13860 hybrid_clock.cc:648] HybridClock initialized: now 1790394420371465 us; error 62 us; skew 500 ppm
I20260926 03:47:00.371903 13860 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:47:00 dist-test-slave-n2th krb5kdc[12649](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394420, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:47:00.375476 13860 init.cc:377] Logged in from keytab as kudu/127.0.194.194@KRBTEST.COM (short username kudu)
I20260926 03:47:00.377616 13860 webserver.cc:460] Webserver started at http://127.0.194.194:33883/ using document root <none> and password file <none>
I20260926 03:47:00.378473 13860 fs_manager.cc:359] Metadata directory not provided
I20260926 03:47:00.378646 13860 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 03:47:00 dist-test-slave-n2th krb5kdc[12649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394420, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.194@KRBTEST.COM for HTTP/127.0.194.212@KRBTEST.COM
I20260926 03:47:00.399466 13860 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.002s sys 0.001s
I20260926 03:47:00.407505 13886 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:47:00.409477 13860 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.004s sys 0.000s
I20260926 03:47:00.409695 13860 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/minicluster-data/ts-1/wal
uuid: "2e24b8bd66f647e98e3147831ef50fbe"
format_stamp: "Formatted at 2026-09-26 03:46:57 on dist-test-slave-n2th"
server_key: "fb7fd2956d758ae515cf9f8e76b1fb9c"
server_key_iv: "6a895db182b5a94a17ca7197df9edf65"
server_key_version: "kuduclusterkey@0"
I20260926 03:47:00.411294 13860 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:47:00.478842 13860 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:47:00.484411 13860 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:47:00.484726 13860 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:47:00.485981 13860 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:47:00.487887 13860 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:47:00.488000 13860 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:47:00.488162 13860 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:47:00.488242 13860 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 03:47:00 dist-test-slave-n2th krb5kdc[12649](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790394420, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:47:00.561964 13860 rpc_server.cc:307] RPC server started. Bound to: 127.0.194.194:35349
I20260926 03:47:00.562043 14000 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.194.194:35349 every 8 connection(s)
I20260926 03:47:00.564182 13860 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/minicluster-data/ts-1/data/info.pb
I20260926 03:47:00.572320 779 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu as pid 13860
I20260926 03:47:00.573138 779 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
/tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.194.212:53233/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.194.195:40201
--local_ip_for_outbound_sockets=127.0.194.195
--tserver_master_addrs=127.0.194.254:40481
--webserver_port=46371
--webserver_interface=127.0.194.195
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/krb5kdc/kudu_127.0.194.195.keytab
--principal=kudu/127.0.194.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.194.212:46639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.194.212:53233/kms with env {KRB5CCNAME=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 03:47:00 dist-test-slave-n2th krb5kdc[12649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394420, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.194@KRBTEST.COM for kudu/127.0.194.254@KRBTEST.COM
I20260926 03:47:00.591799 14003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:00.567315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.194:60789 (local address 127.0.194.254:40481)
0926 03:47:00.567851 (+ 536us) server_negotiation.cc:206] Beginning negotiation
0926 03:47:00.567858 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:47:00.568940 (+ 1082us) server_negotiation.cc:407] Connection header received
0926 03:47:00.571698 (+ 2758us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:47:00.571705 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:47:00.571832 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:47:00.571936 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:47:00.574143 (+ 2207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:47:00.575365 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:47:00.578023 (+ 2658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:47:00.578707 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:47:00.578842 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:47:00.583598 (+ 4756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:47:00.583606 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:47:00.583612 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:47:00.583657 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:47:00.587356 (+ 3699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:47:00.588526 (+ 1170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:47:00.588533 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:47:00.588537 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:47:00.588694 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:47:00.589362 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:47:00.589367 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:47:00.589371 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:47:00.589755 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:47:00.589895 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0926 03:47:00.591133 (+ 1238us) server_negotiation.cc:299] Negotiation successful
0926 03:47:00.591384 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":411,"thread_start_us":274,"threads_started":1}
I20260926 03:47:00.594775 14004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:00.567904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:40481 (local address 127.0.194.194:60789)
0926 03:47:00.568719 (+ 815us) negotiation.cc:107] Waiting for socket to connect
0926 03:47:00.568778 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0926 03:47:00.571293 (+ 2515us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:47:00.572406 (+ 1113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:47:00.572420 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:47:00.573079 (+ 659us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:47:00.573914 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:47:00.573939 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:47:00.575622 (+ 1683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:47:00.575628 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:47:00.577824 (+ 2196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:47:00.577841 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:47:00.579222 (+ 1381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:47:00.579229 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:47:00.579525 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:47:00.580376 (+ 851us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:47:00.580408 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:47:00.583278 (+ 2870us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:47:00.587582 (+ 4304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:47:00.587593 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:47:00.587614 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:47:00.588357 (+ 743us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:47:00.588878 (+ 521us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:47:00.588883 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:47:00.588886 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:47:00.589222 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:47:00.590013 (+ 791us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:47:00.590022 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:47:00.590559 (+ 537us) client_negotiation.cc:769] Sending connection context
0926 03:47:00.592170 (+ 1611us) client_negotiation.cc:240] Negotiation successful
0926 03:47:00.592845 (+ 675us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":296,"threads_started":1}
I20260926 03:47:00.596673 14001 heartbeater.cc:347] Connected to a master server at 127.0.194.254:40481
I20260926 03:47:00.597024 14001 heartbeater.cc:464] Registering TS with master...
I20260926 03:47:00.597934 14001 heartbeater.cc:511] Master 127.0.194.254:40481 requested a full tablet report, sending...
I20260926 03:47:00.600348 13660 ts_manager.cc:194] Registered new tserver with Master: 2e24b8bd66f647e98e3147831ef50fbe (127.0.194.194:35349)
I20260926 03:47:00.602629 13660 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.194.194@KRBTEST.COM'} at 127.0.194.194:60789
I20260926 03:47:00.949235 13857 heartbeater.cc:464] Registering TS with master...
Sep 26 03:47:00 dist-test-slave-n2th krb5kdc[12649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394420, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.254@KRBTEST.COM for HTTP/127.0.194.212@KRBTEST.COM
I20260926 03:47:00.949700 13857 heartbeater.cc:511] Master 127.0.194.254:40481 requested a full tablet report, sending...
I20260926 03:47:00.951045 13660 ts_manager.cc:194] Registered new tserver with Master: 5504b5ac53494038b4744fbdf0a92448 (127.0.194.193:46003)
I20260926 03:47:00.952183 13660 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.194.193@KRBTEST.COM'} at 127.0.194.193:42309
W20260926 03:47:01.043903 14005 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:47:01.044330 14005 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:47:01.044550 14005 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:47:01.056069 14005 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:47:01.056286 14005 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:47:01.056493 14005 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.194.195
I20260926 03:47:01.072180 14005 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.194.212:46639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.194.212:53233/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/krb5kdc/kudu_127.0.194.195.keytab
--principal=kudu/127.0.194.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.194.195:40201
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.194.195
--webserver_port=46371
--tserver_master_addrs=127.0.194.254:40481
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.194.195
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:47:01.073918 14005 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:47:01.075670 14005 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:47:01.093331 14012 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:47:01.093369 14011 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:47:01.103276 14005 server_base.cc:1034] running on GCE node
W20260926 03:47:01.103389 14014 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:47:01.104368 14005 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:47:01.105669 14005 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:47:01.106992 14005 hybrid_clock.cc:648] HybridClock initialized: now 1790394421106886 us; error 57 us; skew 500 ppm
I20260926 03:47:01.107395 14005 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:47:01 dist-test-slave-n2th krb5kdc[12649](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394421, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:47:01.111677 14005 init.cc:377] Logged in from keytab as kudu/127.0.194.195@KRBTEST.COM (short username kudu)
I20260926 03:47:01.113778 14005 webserver.cc:460] Webserver started at http://127.0.194.195:46371/ using document root <none> and password file <none>
I20260926 03:47:01.114854 14005 fs_manager.cc:359] Metadata directory not provided
I20260926 03:47:01.115022 14005 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 03:47:01 dist-test-slave-n2th krb5kdc[12649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394421, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.195@KRBTEST.COM for HTTP/127.0.194.212@KRBTEST.COM
I20260926 03:47:01.168573 14005 fs_manager.cc:709] Time spent opening directory manager: real 0.022s user 0.016s sys 0.003s
I20260926 03:47:01.172669 14022 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:47:01.174615 14005 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 03:47:01.174832 14005 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/minicluster-data/ts-2/wal
uuid: "0f450de5b0ae40d4bb4e3e36ceddbd07"
format_stamp: "Formatted at 2026-09-26 03:46:58 on dist-test-slave-n2th"
server_key: "f0dbb789cf617fc98247f41fdc06f757"
server_key_iv: "dfa67067df09f6963f2dd93fc2e24bc6"
server_key_version: "kuduclusterkey@0"
I20260926 03:47:01.176224 14005 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:47:01.224462 14005 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
Sep 26 03:47:01 dist-test-slave-n2th krb5kdc[12649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394420, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.254@KRBTEST.COM for HTTP/127.0.194.212@KRBTEST.COM
I20260926 03:47:01.233714 14005 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:47:01.234042 14005 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:47:01.244196 14005 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:47:01.246358 14005 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:47:01.246541 14005 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260926 03:47:01.246706 14005 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:47:01.246838 14005 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:47:01.514002 14005 rpc_server.cc:307] RPC server started. Bound to: 127.0.194.195:40201
I20260926 03:47:01.514052 14135 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.194.195:40201 every 8 connection(s)
I20260926 03:47:01.515798 14005 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/minicluster-data/ts-2/data/info.pb
I20260926 03:47:01.525985 779 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu as pid 14005
Sep 26 03:47:01 dist-test-slave-n2th krb5kdc[12649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394421, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.195@KRBTEST.COM for kudu/127.0.194.254@KRBTEST.COM
I20260926 03:47:01.573441 14139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:01.527804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36278 (local address 127.0.194.254:40481)
0926 03:47:01.547168 (+ 19364us) server_negotiation.cc:206] Beginning negotiation
0926 03:47:01.547174 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:47:01.547193 (+ 19us) server_negotiation.cc:407] Connection header received
0926 03:47:01.547276 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:47:01.547282 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:47:01.547387 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:47:01.547510 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:47:01.554236 (+ 6726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:47:01.555400 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:47:01.557455 (+ 2055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:47:01.558120 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:47:01.558271 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:47:01.565394 (+ 7123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:47:01.565402 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:47:01.565407 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:47:01.565454 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:47:01.569082 (+ 3628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:47:01.570049 (+ 967us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:47:01.570056 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:47:01.570060 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:47:01.570232 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:47:01.570708 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:47:01.570713 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:47:01.570716 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:47:01.571389 (+ 673us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:47:01.571642 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0926 03:47:01.572815 (+ 1173us) server_negotiation.cc:299] Negotiation successful
0926 03:47:01.573049 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19252,"thread_start_us":18697,"threads_started":1}
I20260926 03:47:01.582070 14140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:01.521302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.195:42561 (local address 127.0.194.254:40481)
0926 03:47:01.546810 (+ 25508us) server_negotiation.cc:206] Beginning negotiation
0926 03:47:01.546816 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:47:01.551535 (+ 4719us) server_negotiation.cc:407] Connection header received
0926 03:47:01.551602 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:47:01.551607 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:47:01.551699 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:47:01.551821 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:47:01.564391 (+ 12570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:47:01.565925 (+ 1534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:47:01.569035 (+ 3110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:47:01.569680 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:47:01.569776 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:47:01.574219 (+ 4443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:47:01.574227 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:47:01.574231 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:47:01.574276 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:47:01.578499 (+ 4223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:47:01.579458 (+ 959us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:47:01.579463 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:47:01.579467 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:47:01.579594 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:47:01.580070 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:47:01.580075 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:47:01.580079 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:47:01.580379 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:47:01.580530 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0926 03:47:01.581378 (+ 848us) server_negotiation.cc:299] Negotiation successful
0926 03:47:01.581635 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25335,"thread_start_us":34539,"threads_started":1}
I20260926 03:47:01.583174 14138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:01.520249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:40481 (local address 127.0.194.195:42561)
0926 03:47:01.548291 (+ 28042us) negotiation.cc:107] Waiting for socket to connect
0926 03:47:01.548362 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0926 03:47:01.551040 (+ 2678us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:47:01.558875 (+ 7835us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:47:01.558890 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:47:01.559619 (+ 729us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:47:01.560569 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:47:01.560604 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:47:01.566605 (+ 6001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:47:01.566613 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:47:01.568839 (+ 2226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:47:01.568859 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:47:01.570196 (+ 1337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:47:01.570205 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:47:01.570492 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:47:01.571246 (+ 754us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:47:01.571276 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:47:01.573891 (+ 2615us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:47:01.578655 (+ 4764us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:47:01.578678 (+ 23us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:47:01.578692 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:47:01.579326 (+ 634us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:47:01.579717 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:47:01.579722 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:47:01.579725 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:47:01.579944 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:47:01.580509 (+ 565us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:47:01.580517 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:47:01.580970 (+ 453us) client_negotiation.cc:769] Sending connection context
0926 03:47:01.581290 (+ 320us) client_negotiation.cc:240] Negotiation successful
0926 03:47:01.581882 (+ 592us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27651,"thread_start_us":34349,"threads_started":1}
I20260926 03:47:01.584612 14136 heartbeater.cc:347] Connected to a master server at 127.0.194.254:40481
I20260926 03:47:01.584964 14136 heartbeater.cc:464] Registering TS with master...
I20260926 03:47:01.585834 14136 heartbeater.cc:511] Master 127.0.194.254:40481 requested a full tablet report, sending...
I20260926 03:47:01.587666 13660 ts_manager.cc:194] Registered new tserver with Master: 0f450de5b0ae40d4bb4e3e36ceddbd07 (127.0.194.195:40201)
I20260926 03:47:01.588752 13660 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.194.195@KRBTEST.COM'} at 127.0.194.195:42561
I20260926 03:47:01.590246 779 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 03:47:01.608631 14001 heartbeater.cc:503] Master 127.0.194.254:40481 was elected leader, sending a full tablet report...
I20260926 03:47:01.608704 14140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:01.593745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36280 (local address 127.0.194.254:40481)
0926 03:47:01.593952 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0926 03:47:01.593961 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:47:01.594472 (+ 511us) server_negotiation.cc:407] Connection header received
0926 03:47:01.594652 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:47:01.594656 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:47:01.594806 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:47:01.594919 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:47:01.596901 (+ 1982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:47:01.597603 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:47:01.599533 (+ 1930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:47:01.599926 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:47:01.600029 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:47:01.603287 (+ 3258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:47:01.603292 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:47:01.603295 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:47:01.603328 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:47:01.605896 (+ 2568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:47:01.606716 (+ 820us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:47:01.606720 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:47:01.606722 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:47:01.606921 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:47:01.607328 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:47:01.607332 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:47:01.607334 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:47:01.607591 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:47:01.607762 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0926 03:47:01.608152 (+ 390us) server_negotiation.cc:299] Negotiation successful
0926 03:47:01.608379 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 03:47:01.614200 13660 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36280:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 03:47:01.622074 13660 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 03:47:01.651506 14159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:01.636268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.193:46003 (local address 127.0.0.1:52882)
0926 03:47:01.638123 (+ 1855us) negotiation.cc:107] Waiting for socket to connect
0926 03:47:01.638185 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0926 03:47:01.638580 (+ 395us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:47:01.639840 (+ 1260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:47:01.639850 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:47:01.639894 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:47:01.640420 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:47:01.640436 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:47:01.643282 (+ 2846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:47:01.643292 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:47:01.646617 (+ 3325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:47:01.646655 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:47:01.649867 (+ 3212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:47:01.649877 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:47:01.650209 (+ 332us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:47:01.650296 (+ 87us) client_negotiation.cc:769] Sending connection context
0926 03:47:01.650621 (+ 325us) client_negotiation.cc:240] Negotiation successful
0926 03:47:01.650949 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1424,"thread_start_us":1243,"threads_started":1}
I20260926 03:47:01.652741 14158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:01.636247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52882 (local address 127.0.194.193:46003)
0926 03:47:01.636910 (+ 663us) server_negotiation.cc:206] Beginning negotiation
0926 03:47:01.636919 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:47:01.638477 (+ 1558us) server_negotiation.cc:407] Connection header received
0926 03:47:01.638872 (+ 395us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:47:01.638884 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:47:01.639219 (+ 335us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:47:01.639431 (+ 212us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:47:01.640700 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:47:01.643053 (+ 2353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:47:01.646952 (+ 3899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:47:01.649106 (+ 2154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:47:01.649312 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:47:01.649641 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0926 03:47:01.652006 (+ 2365us) server_negotiation.cc:299] Negotiation successful
0926 03:47:01.652283 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":479,"thread_start_us":295,"threads_started":1}
I20260926 03:47:01.659680 13792 tablet_service.cc:1467] Processing CreateTablet for tablet ee5cda4a06d34290b5f34fccc5c6402a (DEFAULT_TABLE table=test-table [id=2d13904c90624d6c9dd67c67d694ab0b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:47:01.661296 13792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee5cda4a06d34290b5f34fccc5c6402a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:47:01.664508 14159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:01.636722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.195:40201 (local address 127.0.0.1:34448)
0926 03:47:01.653336 (+ 16614us) negotiation.cc:107] Waiting for socket to connect
0926 03:47:01.653365 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0926 03:47:01.653525 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:47:01.657721 (+ 4196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:47:01.657724 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:47:01.657744 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:47:01.658086 (+ 342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:47:01.658104 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:47:01.659814 (+ 1710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:47:01.659821 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:47:01.661488 (+ 1667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:47:01.661906 (+ 418us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:47:01.663609 (+ 1703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:47:01.663615 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:47:01.663810 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:47:01.663846 (+ 36us) client_negotiation.cc:769] Sending connection context
0926 03:47:01.663966 (+ 120us) client_negotiation.cc:240] Negotiation successful
0926 03:47:01.664114 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16446,"mutex_wait_us":33,"thread_start_us":21363,"threads_started":1}
I20260926 03:47:01.664644 14161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:01.637206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34448 (local address 127.0.194.195:40201)
0926 03:47:01.657123 (+ 19917us) server_negotiation.cc:206] Beginning negotiation
0926 03:47:01.657131 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:47:01.657167 (+ 36us) server_negotiation.cc:407] Connection header received
0926 03:47:01.657260 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:47:01.657267 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:47:01.657526 (+ 259us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:47:01.657727 (+ 201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:47:01.658324 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:47:01.659667 (+ 1343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:47:01.662069 (+ 2402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:47:01.663434 (+ 1365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:47:01.663593 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:47:01.663828 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0926 03:47:01.664017 (+ 189us) server_negotiation.cc:299] Negotiation successful
0926 03:47:01.664168 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19744,"thread_start_us":19555,"threads_started":1}
I20260926 03:47:01.666091 14163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:01.639139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46416 (local address 127.0.194.194:35349)
0926 03:47:01.654528 (+ 15389us) server_negotiation.cc:206] Beginning negotiation
0926 03:47:01.654541 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 03:47:01.655431 (+ 890us) server_negotiation.cc:407] Connection header received
0926 03:47:01.655582 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:47:01.655594 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:47:01.655939 (+ 345us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:47:01.656194 (+ 255us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:47:01.659076 (+ 2882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:47:01.660577 (+ 1501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:47:01.663043 (+ 2466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:47:01.664641 (+ 1598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:47:01.665150 (+ 509us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:47:01.665402 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0926 03:47:01.665596 (+ 194us) server_negotiation.cc:299] Negotiation successful
0926 03:47:01.665761 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15156,"thread_start_us":14970,"threads_started":1}
I20260926 03:47:01.667491 14162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:01.638603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.194:35349 (local address 127.0.0.1:46416)
0926 03:47:01.655095 (+ 16492us) negotiation.cc:107] Waiting for socket to connect
0926 03:47:01.655119 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 03:47:01.655283 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:47:01.656510 (+ 1227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:47:01.656518 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:47:01.656560 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:47:01.658877 (+ 2317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:47:01.658890 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:47:01.660726 (+ 1836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:47:01.660732 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:47:01.662867 (+ 2135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:47:01.662882 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:47:01.665215 (+ 2333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:47:01.665221 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:47:01.665392 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:47:01.665422 (+ 30us) client_negotiation.cc:769] Sending connection context
0926 03:47:01.665508 (+ 86us) client_negotiation.cc:240] Negotiation successful
0926 03:47:01.665639 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16325,"mutex_wait_us":78,"thread_start_us":16144,"threads_started":1}
I20260926 03:47:01.668593 14071 tablet_service.cc:1467] Processing CreateTablet for tablet ee5cda4a06d34290b5f34fccc5c6402a (DEFAULT_TABLE table=test-table [id=2d13904c90624d6c9dd67c67d694ab0b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:47:01.669804 14071 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee5cda4a06d34290b5f34fccc5c6402a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:47:01.671355 13936 tablet_service.cc:1467] Processing CreateTablet for tablet ee5cda4a06d34290b5f34fccc5c6402a (DEFAULT_TABLE table=test-table [id=2d13904c90624d6c9dd67c67d694ab0b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:47:01.672773 13936 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee5cda4a06d34290b5f34fccc5c6402a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:47:01.678574 14164 tablet_bootstrap.cc:492] T ee5cda4a06d34290b5f34fccc5c6402a P 5504b5ac53494038b4744fbdf0a92448: Bootstrap starting.
I20260926 03:47:01.682417 14164 tablet_bootstrap.cc:654] T ee5cda4a06d34290b5f34fccc5c6402a P 5504b5ac53494038b4744fbdf0a92448: Neither blocks nor log segments found. Creating new log.
I20260926 03:47:01.683969 14165 tablet_bootstrap.cc:492] T ee5cda4a06d34290b5f34fccc5c6402a P 0f450de5b0ae40d4bb4e3e36ceddbd07: Bootstrap starting.
I20260926 03:47:01.684038 14164 log.cc:824] T ee5cda4a06d34290b5f34fccc5c6402a P 5504b5ac53494038b4744fbdf0a92448: Log is configured to *not* fsync() on all Append() calls
I20260926 03:47:01.687631 14165 tablet_bootstrap.cc:654] T ee5cda4a06d34290b5f34fccc5c6402a P 0f450de5b0ae40d4bb4e3e36ceddbd07: Neither blocks nor log segments found. Creating new log.
I20260926 03:47:01.689185 14164 tablet_bootstrap.cc:492] T ee5cda4a06d34290b5f34fccc5c6402a P 5504b5ac53494038b4744fbdf0a92448: No bootstrap required, opened a new log
I20260926 03:47:01.689275 14165 log.cc:824] T ee5cda4a06d34290b5f34fccc5c6402a P 0f450de5b0ae40d4bb4e3e36ceddbd07: Log is configured to *not* fsync() on all Append() calls
I20260926 03:47:01.689477 14164 ts_tablet_manager.cc:1397] T ee5cda4a06d34290b5f34fccc5c6402a P 5504b5ac53494038b4744fbdf0a92448: Time spent bootstrapping tablet: real 0.011s user 0.010s sys 0.000s
I20260926 03:47:01.690588 14167 tablet_bootstrap.cc:492] T ee5cda4a06d34290b5f34fccc5c6402a P 2e24b8bd66f647e98e3147831ef50fbe: Bootstrap starting.
I20260926 03:47:01.692560 14165 tablet_bootstrap.cc:492] T ee5cda4a06d34290b5f34fccc5c6402a P 0f450de5b0ae40d4bb4e3e36ceddbd07: No bootstrap required, opened a new log
I20260926 03:47:01.692811 14165 ts_tablet_manager.cc:1397] T ee5cda4a06d34290b5f34fccc5c6402a P 0f450de5b0ae40d4bb4e3e36ceddbd07: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260926 03:47:01.693559 14167 tablet_bootstrap.cc:654] T ee5cda4a06d34290b5f34fccc5c6402a P 2e24b8bd66f647e98e3147831ef50fbe: Neither blocks nor log segments found. Creating new log.
I20260926 03:47:01.694550 14167 log.cc:824] T ee5cda4a06d34290b5f34fccc5c6402a P 2e24b8bd66f647e98e3147831ef50fbe: Log is configured to *not* fsync() on all Append() calls
I20260926 03:47:01.698307 14167 tablet_bootstrap.cc:492] T ee5cda4a06d34290b5f34fccc5c6402a P 2e24b8bd66f647e98e3147831ef50fbe: No bootstrap required, opened a new log
I20260926 03:47:01.698597 14167 ts_tablet_manager.cc:1397] T ee5cda4a06d34290b5f34fccc5c6402a P 2e24b8bd66f647e98e3147831ef50fbe: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260926 03:47:01.700060 14164 raft_consensus.cc:348] T ee5cda4a06d34290b5f34fccc5c6402a P 5504b5ac53494038b4744fbdf0a92448 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5504b5ac53494038b4744fbdf0a92448" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 46003 } } peers { permanent_uuid: "2e24b8bd66f647e98e3147831ef50fbe" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 35349 } } peers { permanent_uuid: "0f450de5b0ae40d4bb4e3e36ceddbd07" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 40201 } }
I20260926 03:47:01.700527 14164 raft_consensus.cc:374] T ee5cda4a06d34290b5f34fccc5c6402a P 5504b5ac53494038b4744fbdf0a92448 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:47:01.700668 14164 raft_consensus.cc:729] T ee5cda4a06d34290b5f34fccc5c6402a P 5504b5ac53494038b4744fbdf0a92448 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5504b5ac53494038b4744fbdf0a92448, State: Initialized, Role: FOLLOWER
I20260926 03:47:01.701330 14164 consensus_queue.cc:260] T ee5cda4a06d34290b5f34fccc5c6402a P 5504b5ac53494038b4744fbdf0a92448 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5504b5ac53494038b4744fbdf0a92448" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 46003 } } peers { permanent_uuid: "2e24b8bd66f647e98e3147831ef50fbe" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 35349 } } peers { permanent_uuid: "0f450de5b0ae40d4bb4e3e36ceddbd07" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 40201 } }
I20260926 03:47:01.702519 13857 heartbeater.cc:503] Master 127.0.194.254:40481 was elected leader, sending a full tablet report...
I20260926 03:47:01.703220 14164 ts_tablet_manager.cc:1428] T ee5cda4a06d34290b5f34fccc5c6402a P 5504b5ac53494038b4744fbdf0a92448: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260926 03:47:01.703366 14165 raft_consensus.cc:348] T ee5cda4a06d34290b5f34fccc5c6402a P 0f450de5b0ae40d4bb4e3e36ceddbd07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5504b5ac53494038b4744fbdf0a92448" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 46003 } } peers { permanent_uuid: "2e24b8bd66f647e98e3147831ef50fbe" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 35349 } } peers { permanent_uuid: "0f450de5b0ae40d4bb4e3e36ceddbd07" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 40201 } }
I20260926 03:47:01.703802 14165 raft_consensus.cc:374] T ee5cda4a06d34290b5f34fccc5c6402a P 0f450de5b0ae40d4bb4e3e36ceddbd07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:47:01.703926 14165 raft_consensus.cc:729] T ee5cda4a06d34290b5f34fccc5c6402a P 0f450de5b0ae40d4bb4e3e36ceddbd07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f450de5b0ae40d4bb4e3e36ceddbd07, State: Initialized, Role: FOLLOWER
I20260926 03:47:01.704504 14165 consensus_queue.cc:260] T ee5cda4a06d34290b5f34fccc5c6402a P 0f450de5b0ae40d4bb4e3e36ceddbd07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5504b5ac53494038b4744fbdf0a92448" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 46003 } } peers { permanent_uuid: "2e24b8bd66f647e98e3147831ef50fbe" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 35349 } } peers { permanent_uuid: "0f450de5b0ae40d4bb4e3e36ceddbd07" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 40201 } }
I20260926 03:47:01.707551 14136 heartbeater.cc:503] Master 127.0.194.254:40481 was elected leader, sending a full tablet report...
I20260926 03:47:01.707993 14167 raft_consensus.cc:348] T ee5cda4a06d34290b5f34fccc5c6402a P 2e24b8bd66f647e98e3147831ef50fbe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5504b5ac53494038b4744fbdf0a92448" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 46003 } } peers { permanent_uuid: "2e24b8bd66f647e98e3147831ef50fbe" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 35349 } } peers { permanent_uuid: "0f450de5b0ae40d4bb4e3e36ceddbd07" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 40201 } }
I20260926 03:47:01.708495 14167 raft_consensus.cc:374] T ee5cda4a06d34290b5f34fccc5c6402a P 2e24b8bd66f647e98e3147831ef50fbe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:47:01.708513 14170 raft_consensus.cc:482] T ee5cda4a06d34290b5f34fccc5c6402a P 5504b5ac53494038b4744fbdf0a92448 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 03:47:01.708695 14167 raft_consensus.cc:729] T ee5cda4a06d34290b5f34fccc5c6402a P 2e24b8bd66f647e98e3147831ef50fbe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e24b8bd66f647e98e3147831ef50fbe, State: Initialized, Role: FOLLOWER
I20260926 03:47:01.708791 14170 raft_consensus.cc:504] T ee5cda4a06d34290b5f34fccc5c6402a P 5504b5ac53494038b4744fbdf0a92448 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5504b5ac53494038b4744fbdf0a92448" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 46003 } } peers { permanent_uuid: "2e24b8bd66f647e98e3147831ef50fbe" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 35349 } } peers { permanent_uuid: "0f450de5b0ae40d4bb4e3e36ceddbd07" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 40201 } }
I20260926 03:47:01.709497 14165 ts_tablet_manager.cc:1428] T ee5cda4a06d34290b5f34fccc5c6402a P 0f450de5b0ae40d4bb4e3e36ceddbd07: Time spent starting tablet: real 0.017s user 0.016s sys 0.000s
I20260926 03:47:01.709513 14167 consensus_queue.cc:260] T ee5cda4a06d34290b5f34fccc5c6402a P 2e24b8bd66f647e98e3147831ef50fbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5504b5ac53494038b4744fbdf0a92448" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 46003 } } peers { permanent_uuid: "2e24b8bd66f647e98e3147831ef50fbe" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 35349 } } peers { permanent_uuid: "0f450de5b0ae40d4bb4e3e36ceddbd07" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 40201 } }
I20260926 03:47:01.711440 14167 ts_tablet_manager.cc:1428] T ee5cda4a06d34290b5f34fccc5c6402a P 2e24b8bd66f647e98e3147831ef50fbe: Time spent starting tablet: real 0.013s user 0.012s sys 0.001s
I20260926 03:47:01.715541 14170 leader_election.cc:288] T ee5cda4a06d34290b5f34fccc5c6402a P 5504b5ac53494038b4744fbdf0a92448 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2e24b8bd66f647e98e3147831ef50fbe (127.0.194.194:35349), 0f450de5b0ae40d4bb4e3e36ceddbd07 (127.0.194.195:40201)
I20260926 03:47:01.722172 14173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:01.711422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.194:35349 (local address 127.0.194.193:44905)
0926 03:47:01.712732 (+ 1310us) negotiation.cc:107] Waiting for socket to connect
0926 03:47:01.712752 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 03:47:01.713063 (+ 311us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:47:01.713908 (+ 845us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:47:01.713913 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:47:01.713943 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:47:01.714463 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:47:01.714475 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:47:01.717139 (+ 2664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:47:01.717144 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:47:01.719398 (+ 2254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:47:01.719416 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:47:01.721160 (+ 1744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:47:01.721166 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:47:01.721358 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:47:01.721404 (+ 46us) client_negotiation.cc:769] Sending connection context
0926 03:47:01.721507 (+ 103us) client_negotiation.cc:240] Negotiation successful
0926 03:47:01.721648 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1120,"mutex_wait_us":36,"thread_start_us":856,"threads_started":1}
I20260926 03:47:01.722884 14163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:01.711093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.193:44905 (local address 127.0.194.194:35349)
0926 03:47:01.711485 (+ 392us) server_negotiation.cc:206] Beginning negotiation
0926 03:47:01.711492 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:47:01.712887 (+ 1395us) server_negotiation.cc:407] Connection header received
0926 03:47:01.713311 (+ 424us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:47:01.713352 (+ 41us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:47:01.713463 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:47:01.713562 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:47:01.714630 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:47:01.716774 (+ 2144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:47:01.719575 (+ 2801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:47:01.720734 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:47:01.720885 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:47:01.720966 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0926 03:47:01.722411 (+ 1445us) server_negotiation.cc:299] Negotiation successful
0926 03:47:01.722563 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":205}
I20260926 03:47:01.722914 14161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:01.713710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.193:36673 (local address 127.0.194.195:40201)
0926 03:47:01.713922 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0926 03:47:01.713930 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:47:01.715627 (+ 1697us) server_negotiation.cc:407] Connection header received
0926 03:47:01.715948 (+ 321us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:47:01.715955 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:47:01.716029 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:47:01.716091 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:47:01.716885 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:47:01.717933 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:47:01.720487 (+ 2554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:47:01.721628 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:47:01.721770 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:47:01.721844 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 03:47:01.722471 (+ 627us) server_negotiation.cc:299] Negotiation successful
0926 03:47:01.722617 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260926 03:47:01.723628 14174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:01.713223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.195:40201 (local address 127.0.194.193:36673)
0926 03:47:01.715528 (+ 2305us) negotiation.cc:107] Waiting for socket to connect
0926 03:47:01.715549 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 03:47:01.715727 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:47:01.716277 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:47:01.716282 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:47:01.716311 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:47:01.716624 (+ 313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:47:01.716638 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:47:01.718162 (+ 1524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:47:01.718167 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:47:01.720291 (+ 2124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:47:01.720306 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:47:01.722081 (+ 1775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:47:01.722088 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:47:01.722279 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:47:01.722313 (+ 34us) client_negotiation.cc:769] Sending connection context
0926 03:47:01.723147 (+ 834us) client_negotiation.cc:240] Negotiation successful
0926 03:47:01.723300 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2162,"thread_start_us":1982,"threads_started":1}
I20260926 03:47:01.724138 13956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee5cda4a06d34290b5f34fccc5c6402a" candidate_uuid: "5504b5ac53494038b4744fbdf0a92448" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e24b8bd66f647e98e3147831ef50fbe" is_pre_election: true
I20260926 03:47:01.724617 14091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee5cda4a06d34290b5f34fccc5c6402a" candidate_uuid: "5504b5ac53494038b4744fbdf0a92448" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f450de5b0ae40d4bb4e3e36ceddbd07" is_pre_election: true
I20260926 03:47:01.724750 13956 raft_consensus.cc:2454] T ee5cda4a06d34290b5f34fccc5c6402a P 2e24b8bd66f647e98e3147831ef50fbe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5504b5ac53494038b4744fbdf0a92448 in term 0.
I20260926 03:47:01.725564 14091 raft_consensus.cc:2454] T ee5cda4a06d34290b5f34fccc5c6402a P 0f450de5b0ae40d4bb4e3e36ceddbd07 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5504b5ac53494038b4744fbdf0a92448 in term 0.
I20260926 03:47:01.725970 13745 leader_election.cc:302] T ee5cda4a06d34290b5f34fccc5c6402a P 5504b5ac53494038b4744fbdf0a92448 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e24b8bd66f647e98e3147831ef50fbe, 5504b5ac53494038b4744fbdf0a92448; no voters:
I20260926 03:47:01.726475 14170 raft_consensus.cc:2781] T ee5cda4a06d34290b5f34fccc5c6402a P 5504b5ac53494038b4744fbdf0a92448 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 03:47:01.726608 14170 raft_consensus.cc:482] T ee5cda4a06d34290b5f34fccc5c6402a P 5504b5ac53494038b4744fbdf0a92448 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 03:47:01.726725 14170 raft_consensus.cc:3037] T ee5cda4a06d34290b5f34fccc5c6402a P 5504b5ac53494038b4744fbdf0a92448 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:47:01.728998 14170 raft_consensus.cc:504] T ee5cda4a06d34290b5f34fccc5c6402a P 5504b5ac53494038b4744fbdf0a92448 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5504b5ac53494038b4744fbdf0a92448" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 46003 } } peers { permanent_uuid: "2e24b8bd66f647e98e3147831ef50fbe" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 35349 } } peers { permanent_uuid: "0f450de5b0ae40d4bb4e3e36ceddbd07" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 40201 } }
I20260926 03:47:01.729647 14170 leader_election.cc:288] T ee5cda4a06d34290b5f34fccc5c6402a P 5504b5ac53494038b4744fbdf0a92448 [CANDIDATE]: Term 1 election: Requested vote from peers 2e24b8bd66f647e98e3147831ef50fbe (127.0.194.194:35349), 0f450de5b0ae40d4bb4e3e36ceddbd07 (127.0.194.195:40201)
I20260926 03:47:01.730050 13956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee5cda4a06d34290b5f34fccc5c6402a" candidate_uuid: "5504b5ac53494038b4744fbdf0a92448" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e24b8bd66f647e98e3147831ef50fbe"
I20260926 03:47:01.730178 14091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee5cda4a06d34290b5f34fccc5c6402a" candidate_uuid: "5504b5ac53494038b4744fbdf0a92448" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f450de5b0ae40d4bb4e3e36ceddbd07"
I20260926 03:47:01.730268 13956 raft_consensus.cc:3037] T ee5cda4a06d34290b5f34fccc5c6402a P 2e24b8bd66f647e98e3147831ef50fbe [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:47:01.730391 14091 raft_consensus.cc:3037] T ee5cda4a06d34290b5f34fccc5c6402a P 0f450de5b0ae40d4bb4e3e36ceddbd07 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:47:01.733551 13956 raft_consensus.cc:2454] T ee5cda4a06d34290b5f34fccc5c6402a P 2e24b8bd66f647e98e3147831ef50fbe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5504b5ac53494038b4744fbdf0a92448 in term 1.
I20260926 03:47:01.733551 14091 raft_consensus.cc:2454] T ee5cda4a06d34290b5f34fccc5c6402a P 0f450de5b0ae40d4bb4e3e36ceddbd07 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5504b5ac53494038b4744fbdf0a92448 in term 1.
I20260926 03:47:01.734030 13745 leader_election.cc:302] T ee5cda4a06d34290b5f34fccc5c6402a P 5504b5ac53494038b4744fbdf0a92448 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0f450de5b0ae40d4bb4e3e36ceddbd07, 2e24b8bd66f647e98e3147831ef50fbe, 5504b5ac53494038b4744fbdf0a92448; no voters:
I20260926 03:47:01.734335 14170 raft_consensus.cc:2781] T ee5cda4a06d34290b5f34fccc5c6402a P 5504b5ac53494038b4744fbdf0a92448 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:47:01.734840 14170 raft_consensus.cc:686] T ee5cda4a06d34290b5f34fccc5c6402a P 5504b5ac53494038b4744fbdf0a92448 [term 1 LEADER]: Becoming Leader. State: Replica: 5504b5ac53494038b4744fbdf0a92448, State: Running, Role: LEADER
I20260926 03:47:01.735323 14170 consensus_queue.cc:237] T ee5cda4a06d34290b5f34fccc5c6402a P 5504b5ac53494038b4744fbdf0a92448 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5504b5ac53494038b4744fbdf0a92448" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 46003 } } peers { permanent_uuid: "2e24b8bd66f647e98e3147831ef50fbe" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 35349 } } peers { permanent_uuid: "0f450de5b0ae40d4bb4e3e36ceddbd07" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 40201 } }
I20260926 03:47:01.740221 13660 catalog_manager.cc:5495] T ee5cda4a06d34290b5f34fccc5c6402a P 5504b5ac53494038b4744fbdf0a92448 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5504b5ac53494038b4744fbdf0a92448 (127.0.194.193). New cstate: current_term: 1 leader_uuid: "5504b5ac53494038b4744fbdf0a92448" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5504b5ac53494038b4744fbdf0a92448" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 46003 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2e24b8bd66f647e98e3147831ef50fbe" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 35349 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0f450de5b0ae40d4bb4e3e36ceddbd07" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 40201 } health_report { overall_health: UNKNOWN } } }
I20260926 03:47:01.765358 14158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:01.759331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52890 (local address 127.0.194.193:46003)
0926 03:47:01.759638 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0926 03:47:01.759647 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:47:01.759670 (+ 23us) server_negotiation.cc:407] Connection header received
0926 03:47:01.760385 (+ 715us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:47:01.760390 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:47:01.760465 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:47:01.760618 (+ 153us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:47:01.761020 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:47:01.761734 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:47:01.763157 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:47:01.763555 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:47:01.763697 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:47:01.764273 (+ 576us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:47:01.764774 (+ 501us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:47:01.764932 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0926 03:47:01.764992 (+ 60us) server_negotiation.cc:299] Negotiation successful
0926 03:47:01.765072 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162}
I20260926 03:47:01.773617 14091 raft_consensus.cc:1261] T ee5cda4a06d34290b5f34fccc5c6402a P 0f450de5b0ae40d4bb4e3e36ceddbd07 [term 1 FOLLOWER]: Refusing update from remote peer 5504b5ac53494038b4744fbdf0a92448: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:47:01.773618 13956 raft_consensus.cc:1261] T ee5cda4a06d34290b5f34fccc5c6402a P 2e24b8bd66f647e98e3147831ef50fbe [term 1 FOLLOWER]: Refusing update from remote peer 5504b5ac53494038b4744fbdf0a92448: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:47:01.774596 14175 consensus_queue.cc:1035] T ee5cda4a06d34290b5f34fccc5c6402a P 5504b5ac53494038b4744fbdf0a92448 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e24b8bd66f647e98e3147831ef50fbe" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 35349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:47:01.775074 14170 consensus_queue.cc:1035] T ee5cda4a06d34290b5f34fccc5c6402a P 5504b5ac53494038b4744fbdf0a92448 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f450de5b0ae40d4bb4e3e36ceddbd07" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 40201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:47:01.788373 14180 mvcc.cc:205] Tried to move back new op lower bound from 7333455551575105536 to 7333455551432298496. Current Snapshot: MvccSnapshot[applied={T|T < 7333455551575105536}]
I20260926 03:47:01.801262 13660 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36280:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:47:01.801717 13660 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36280:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:47:01.808314 13660 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b27896b679d54f50998598522e611d7c: Sending DeleteTablet for 3 replicas of tablet ee5cda4a06d34290b5f34fccc5c6402a
I20260926 03:47:01.809821 13792 tablet_service.cc:1514] Processing DeleteTablet for tablet ee5cda4a06d34290b5f34fccc5c6402a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:47:01 UTC) from {username='kudu', principal='kudu/127.0.194.254@KRBTEST.COM'} at 127.0.0.1:52882
I20260926 03:47:01.810514 14071 tablet_service.cc:1514] Processing DeleteTablet for tablet ee5cda4a06d34290b5f34fccc5c6402a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:47:01 UTC) from {username='kudu', principal='kudu/127.0.194.254@KRBTEST.COM'} at 127.0.0.1:34448
I20260926 03:47:01.815740 13935 tablet_service.cc:1514] Processing DeleteTablet for tablet ee5cda4a06d34290b5f34fccc5c6402a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:47:01 UTC) from {username='kudu', principal='kudu/127.0.194.254@KRBTEST.COM'} at 127.0.0.1:46416
I20260926 03:47:01.816425 13936 tablet_service.cc:1514] Processing DeleteTablet for tablet ee5cda4a06d34290b5f34fccc5c6402a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:47:01 UTC) from {username='kudu', principal='kudu/127.0.194.254@KRBTEST.COM'} at 127.0.0.1:46416
I20260926 03:47:01.817022 779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu pid 13697
I20260926 03:47:01.823290 14190 tablet_replica.cc:330] T ee5cda4a06d34290b5f34fccc5c6402a P 2e24b8bd66f647e98e3147831ef50fbe: stopping tablet replica
I20260926 03:47:01.824268 14190 raft_consensus.cc:2229] T ee5cda4a06d34290b5f34fccc5c6402a P 2e24b8bd66f647e98e3147831ef50fbe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:47:01.825505 14190 raft_consensus.cc:2258] T ee5cda4a06d34290b5f34fccc5c6402a P 2e24b8bd66f647e98e3147831ef50fbe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:47:01.829128 14190 ts_tablet_manager.cc:1905] T ee5cda4a06d34290b5f34fccc5c6402a P 2e24b8bd66f647e98e3147831ef50fbe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:47:01.830317 14189 tablet_replica.cc:330] T ee5cda4a06d34290b5f34fccc5c6402a P 0f450de5b0ae40d4bb4e3e36ceddbd07: stopping tablet replica
I20260926 03:47:01.831735 14189 raft_consensus.cc:2229] T ee5cda4a06d34290b5f34fccc5c6402a P 0f450de5b0ae40d4bb4e3e36ceddbd07 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:47:01.832489 14189 raft_consensus.cc:2258] T ee5cda4a06d34290b5f34fccc5c6402a P 0f450de5b0ae40d4bb4e3e36ceddbd07 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:47:01.836270 14189 ts_tablet_manager.cc:1905] T ee5cda4a06d34290b5f34fccc5c6402a P 0f450de5b0ae40d4bb4e3e36ceddbd07: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:47:01.838037 14190 ts_tablet_manager.cc:1918] T ee5cda4a06d34290b5f34fccc5c6402a P 2e24b8bd66f647e98e3147831ef50fbe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:47:01.838276 14190 log.cc:1196] T ee5cda4a06d34290b5f34fccc5c6402a P 2e24b8bd66f647e98e3147831ef50fbe: Deleting WAL directory at /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/minicluster-data/ts-1/wal/wals/ee5cda4a06d34290b5f34fccc5c6402a
I20260926 03:47:01.838991 14190 ts_tablet_manager.cc:1939] T ee5cda4a06d34290b5f34fccc5c6402a P 2e24b8bd66f647e98e3147831ef50fbe: Deleting consensus metadata
I20260926 03:47:01.841051 13645 catalog_manager.cc:4841] TS 2e24b8bd66f647e98e3147831ef50fbe (127.0.194.194:35349): tablet ee5cda4a06d34290b5f34fccc5c6402a (table test-table [id=2d13904c90624d6c9dd67c67d694ab0b]) successfully deleted
W20260926 03:47:01.841800 13645 catalog_manager.cc:4808] TS 2e24b8bd66f647e98e3147831ef50fbe (127.0.194.194:35349): delete failed for tablet ee5cda4a06d34290b5f34fccc5c6402a because the tablet was not found. No further retry: Not found: Tablet not found: ee5cda4a06d34290b5f34fccc5c6402a
I20260926 03:47:01.847091 14189 ts_tablet_manager.cc:1918] T ee5cda4a06d34290b5f34fccc5c6402a P 0f450de5b0ae40d4bb4e3e36ceddbd07: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:47:01.847370 14189 log.cc:1196] T ee5cda4a06d34290b5f34fccc5c6402a P 0f450de5b0ae40d4bb4e3e36ceddbd07: Deleting WAL directory at /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790394221564982-779-0/minicluster-data/ts-2/wal/wals/ee5cda4a06d34290b5f34fccc5c6402a
I20260926 03:47:01.847980 14189 ts_tablet_manager.cc:1939] T ee5cda4a06d34290b5f34fccc5c6402a P 0f450de5b0ae40d4bb4e3e36ceddbd07: Deleting consensus metadata
I20260926 03:47:01.849912 13646 catalog_manager.cc:4841] TS 0f450de5b0ae40d4bb4e3e36ceddbd07 (127.0.194.195:40201): tablet ee5cda4a06d34290b5f34fccc5c6402a (table test-table [id=2d13904c90624d6c9dd67c67d694ab0b]) successfully deleted
I20260926 03:47:01.850332 14158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:01.820165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52906 (local address 127.0.194.193:46003)
0926 03:47:01.820403 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0926 03:47:01.820411 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:47:01.820435 (+ 24us) server_negotiation.cc:407] Connection header received
0926 03:47:01.820511 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:47:01.820519 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:47:01.820652 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:47:01.820778 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:47:01.828191 (+ 7413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:47:01.829379 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:47:01.831775 (+ 2396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:47:01.832531 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:47:01.832679 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:47:01.837452 (+ 4773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:47:01.837468 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:47:01.837495 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:47:01.837543 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:47:01.843865 (+ 6322us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:47:01.845221 (+ 1356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:47:01.845236 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:47:01.845243 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:47:01.845503 (+ 260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:47:01.846067 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:47:01.846079 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:47:01.846085 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:47:01.846776 (+ 691us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:47:01.847207 (+ 431us) server_negotiation.cc:1035] Waiting for connection context
0926 03:47:01.848863 (+ 1656us) server_negotiation.cc:299] Negotiation successful
0926 03:47:01.849230 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260926 03:47:01.850507 14188 tablet_replica.cc:330] T ee5cda4a06d34290b5f34fccc5c6402a P 5504b5ac53494038b4744fbdf0a92448: stopping tablet replica
I20260926 03:47:01.851310 13852 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:47:01.851431 14188 raft_consensus.cc:2229] T ee5cda4a06d34290b5f34fccc5c6402a P 5504b5ac53494038b4744fbdf0a92448 [term 1 LEADER]: Raft consensus shutting down.
I20260926 03:47:01.953280 14188 raft_consensus.cc:2258] T ee5cda4a06d34290b5f34fccc5c6402a P 5504b5ac53494038b4744fbdf0a92448 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:47:01.955693 14188 ts_tablet_manager.cc:1905] T ee5cda4a06d34290b5f34fccc5c6402a P 5504b5ac53494038b4744fbdf0a92448: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:47:01.958927 779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu with pid 13697
W20260926 03:47:01.977798 13645 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.194.193:46003 (error 108)
I20260926 03:47:01.978576 779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu pid 13860
I20260926 03:47:01.979794 14162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:01.979320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.193:46003 (local address 127.0.0.1:52922)
0926 03:47:01.979488 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0926 03:47:01.979600 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.193:46003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260926 03:47:01.980034 13645 catalog_manager.cc:4584] TS 5504b5ac53494038b4744fbdf0a92448 (127.0.194.193:46003): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ee5cda4a06d34290b5f34fccc5c6402a: Network error: Client connection negotiation failed: client connection to 127.0.194.193:46003: connect: Connection refused (error 111)
I20260926 03:47:01.992913 14163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:01.979531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46432 (local address 127.0.194.194:35349)
0926 03:47:01.979722 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0926 03:47:01.979726 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:47:01.979746 (+ 20us) server_negotiation.cc:407] Connection header received
0926 03:47:01.979945 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:47:01.979950 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:47:01.980042 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:47:01.980165 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:47:01.981599 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:47:01.982528 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:47:01.983970 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:47:01.984385 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:47:01.984491 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:47:01.987168 (+ 2677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:47:01.987181 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:47:01.987197 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:47:01.987229 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:47:01.990338 (+ 3109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:47:01.991173 (+ 835us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:47:01.991181 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:47:01.991183 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:47:01.991316 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:47:01.991636 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:47:01.991639 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:47:01.991641 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:47:01.991983 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:47:01.992213 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0926 03:47:01.992470 (+ 257us) server_negotiation.cc:299] Negotiation successful
0926 03:47:01.992632 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 03:47:01.993618 13996 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:47:02.033891 14162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:02.033333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.193:46003 (local address 127.0.0.1:52934)
0926 03:47:02.033567 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0926 03:47:02.033690 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.193:46003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260926 03:47:02.035167 14162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:02.034713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.193:46003 (local address 127.0.0.1:52942)
0926 03:47:02.034931 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0926 03:47:02.035031 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.193:46003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
I20260926 03:47:02.084802 779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu with pid 13860
I20260926 03:47:02.095852 14162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:02.095294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.193:46003 (local address 127.0.0.1:52958)
0926 03:47:02.095527 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0926 03:47:02.095644 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.193:46003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260926 03:47:02.097002 14162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:02.096575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.193:46003 (local address 127.0.0.1:52972)
0926 03:47:02.096733 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0926 03:47:02.096835 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.193:46003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260926 03:47:02.103206 779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu pid 14005
I20260926 03:47:02.117405 14161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:02.103988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34452 (local address 127.0.194.195:40201)
0926 03:47:02.104193 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0926 03:47:02.104200 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:47:02.104303 (+ 103us) server_negotiation.cc:407] Connection header received
0926 03:47:02.104520 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:47:02.104532 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:47:02.104607 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:47:02.104725 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:47:02.106030 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:47:02.106852 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:47:02.108142 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:47:02.108541 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:47:02.108688 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:47:02.111588 (+ 2900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:47:02.111603 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:47:02.111619 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:47:02.111661 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:47:02.114440 (+ 2779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:47:02.115235 (+ 795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:47:02.115252 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:47:02.115254 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:47:02.115400 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:47:02.115808 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:47:02.115815 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:47:02.115818 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:47:02.116188 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:47:02.116501 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0926 03:47:02.116832 (+ 331us) server_negotiation.cc:299] Negotiation successful
0926 03:47:02.117057 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 03:47:02.118105 14131 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:47:02.178313 14162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:02.177831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.193:46003 (local address 127.0.0.1:52988)
0926 03:47:02.178044 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0926 03:47:02.178140 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.193:46003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260926 03:47:02.179440 14162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:02.179052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.193:46003 (local address 127.0.0.1:52994)
0926 03:47:02.179243 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0926 03:47:02.179323 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.193:46003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260926 03:47:02.207877 779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu with pid 14005
I20260926 03:47:02.224536 779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu pid 13627
I20260926 03:47:02.225764 13690 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:47:02.337981 779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu with pid 13627
I20260926 03:47:02.362861 779 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260926 03:47:02.698360 779 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260926 03:47:02.698591 779 mini_ranger.cc:66] Stopping Ranger...
I20260926 03:47:03.044340 779 mini_ranger.cc:68] Stopped Ranger
2026-09-26 03:47:03.044 UTC [12679] LOG: received smart shutdown request
2026-09-26 03:47:03.047 UTC [12679] LOG: background worker "logical replication launcher" (PID 12688) exited with exit code 1
2026-09-26 03:47:03.048 UTC [12683] LOG: shutting down
2026-09-26 03:47:03.206 UTC [12679] LOG: database system is shut down
2026-09-26T03:47:03Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (53347 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790394221564982-779-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:47:03 dist-test-slave-n2th krb5kdc[14204](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:47:03 dist-test-slave-n2th krb5kdc[14204](info): set up 2 sockets
Sep 26 03:47:03 dist-test-slave-n2th krb5kdc[14204](info): commencing operation
krb5kdc: starting...
W20260926 03:47:04.435189 779 mini_kdc.cc:121] Time spent starting KDC: real 1.062s user 0.002s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790394221564982-779-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790394221564982-779-0/krb5kdc/test-user.keytab.
Sep 26 03:47:04 dist-test-slave-n2th krb5kdc[14204](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394424, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:47:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:47:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.194.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.194.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.194.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790394221564982-779-0/krb5kdc/kudu_127.0.194.254.keytab.
Entry for principal kudu/127.0.194.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790394221564982-779-0/krb5kdc/kudu_127.0.194.254.keytab.
I20260926 03:47:04.676143 779 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
/tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790394221564982-779-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790394221564982-779-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790394221564982-779-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790394221564982-779-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.194.254:41069
--webserver_interface=127.0.194.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790394221564982-779-0/krb5kdc/kudu_127.0.194.254.keytab
--principal=kudu/127.0.194.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.194.212:36299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.194.254:41069,127.0.194.253:45537,127.0.194.252:34233
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790394221564982-779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790394221564982-779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790394221564982-779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:47:05.012688 14217 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:47:05.013022 14217 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:47:05.013114 14217 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:47:05.020622 14217 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:47:05.020735 14217 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:47:05.020789 14217 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:47:05.020843 14217 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:47:05.020880 14217 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:47:05.031056 14217 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.194.212:36299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790394221564982-779-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790394221564982-779-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.194.254:41069,127.0.194.253:45537,127.0.194.252:34233
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790394221564982-779-0/krb5kdc/kudu_127.0.194.254.keytab
--principal=kudu/127.0.194.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.194.254:41069
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790394221564982-779-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.194.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790394221564982-779-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:47:05.032361 14217 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:47:05.033649 14217 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:47:05.045877 14222 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:47:05.045900 14223 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:47:05.046195 14225 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:47:05.046820 14217 server_base.cc:1034] running on GCE node
I20260926 03:47:05.047669 14217 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:47:05.049247 14217 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:47:05.050494 14217 hybrid_clock.cc:648] HybridClock initialized: now 1790394425050441 us; error 49 us; skew 500 ppm
I20260926 03:47:05.050925 14217 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:47:05 dist-test-slave-n2th krb5kdc[14204](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394425, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:47:05.063086 14217 init.cc:377] Logged in from keytab as kudu/127.0.194.254@KRBTEST.COM (short username kudu)
I20260926 03:47:05.064852 14217 webserver.cc:460] Webserver started at http://127.0.194.254:34205/ using document root <none> and password file <none>
I20260926 03:47:05.065644 14217 fs_manager.cc:359] Metadata directory not provided
I20260926 03:47:05.065814 14217 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:47:05.066148 14217 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:47:05.069474 14217 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790394221564982-779-0/minicluster-data/master-0/data/instance:
uuid: "a42cb64b376e49b6af88f02210fb9d2d"
format_stamp: "Formatted at 2026-09-26 03:47:05 on dist-test-slave-n2th"
server_key: "380eb90ef892215942e028860a54b0dc"
server_key_iv: "103fa5efe3183d9b5cb5e9b52e2eec8a"
server_key_version: "encryptionkey@0"
I20260926 03:47:05.070307 14217 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790394221564982-779-0/minicluster-data/master-0/wal/instance:
uuid: "a42cb64b376e49b6af88f02210fb9d2d"
format_stamp: "Formatted at 2026-09-26 03:47:05 on dist-test-slave-n2th"
server_key: "380eb90ef892215942e028860a54b0dc"
server_key_iv: "103fa5efe3183d9b5cb5e9b52e2eec8a"
server_key_version: "encryptionkey@0"
I20260926 03:47:05.075857 14217 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260926 03:47:05.079579 14232 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:47:05.081053 14217 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260926 03:47:05.081247 14217 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790394221564982-779-0/minicluster-data/master-0/data,/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790394221564982-779-0/minicluster-data/master-0/wal
uuid: "a42cb64b376e49b6af88f02210fb9d2d"
format_stamp: "Formatted at 2026-09-26 03:47:05 on dist-test-slave-n2th"
server_key: "380eb90ef892215942e028860a54b0dc"
server_key_iv: "103fa5efe3183d9b5cb5e9b52e2eec8a"
server_key_version: "encryptionkey@0"
I20260926 03:47:05.081478 14217 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790394221564982-779-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790394221564982-779-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790394221564982-779-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:47:05.115285 14217 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:47:05.120081 14217 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:47:05.120389 14217 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:47:05.139626 14217 rpc_server.cc:307] RPC server started. Bound to: 127.0.194.254:41069
I20260926 03:47:05.139665 14283 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.194.254:41069 every 8 connection(s)
I20260926 03:47:05.141119 14217 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790394221564982-779-0/minicluster-data/master-0/data/info.pb
I20260926 03:47:05.144956 14284 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:47:05.150393 779 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu as pid 14217
I20260926 03:47:05.150632 779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790394221564982-779-0/minicluster-data/master-0/wal/instance
I20260926 03:47:05.151305 779 external_mini_cluster.cc:1428] Setting key 12249324d2b80b7368ca02ac207e9af6
I20260926 03:47:05.151881 14284 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.194.254" port: 41069 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 03:47:05 dist-test-slave-n2th krb5kdc[14204](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394425, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.254@KRBTEST.COM for kudu/127.0.194.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.194.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.194.253@KRBTEST.COM" created.
I20260926 03:47:05.176584 14286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:05.155366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:41069 (local address 127.0.0.1:57980)
0926 03:47:05.156457 (+ 1091us) negotiation.cc:107] Waiting for socket to connect
0926 03:47:05.156522 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0926 03:47:05.158516 (+ 1994us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:47:05.159431 (+ 915us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:47:05.159438 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:47:05.160217 (+ 779us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:47:05.160567 (+ 350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:47:05.160585 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:47:05.162182 (+ 1597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:47:05.162191 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:47:05.163361 (+ 1170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:47:05.163375 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:47:05.164636 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:47:05.164643 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:47:05.164818 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:47:05.165566 (+ 748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:47:05.165590 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:47:05.168090 (+ 2500us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:47:05.171738 (+ 3648us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:47:05.171750 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:47:05.171761 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:47:05.172251 (+ 490us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:47:05.172638 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:47:05.172641 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:47:05.172644 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:47:05.172801 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:47:05.173840 (+ 1039us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:47:05.173851 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:47:05.174115 (+ 264us) client_negotiation.cc:769] Sending connection context
0926 03:47:05.174369 (+ 254us) client_negotiation.cc:240] Negotiation successful
0926 03:47:05.175097 (+ 728us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":732,"mutex_wait_us":208,"thread_start_us":467,"threads_started":1}
I20260926 03:47:05.176579 14287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:05.155349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57980 (local address 127.0.194.254:41069)
0926 03:47:05.157175 (+ 1826us) server_negotiation.cc:206] Beginning negotiation
0926 03:47:05.157185 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 03:47:05.157240 (+ 55us) server_negotiation.cc:407] Connection header received
0926 03:47:05.158950 (+ 1710us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:47:05.158955 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:47:05.159199 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:47:05.159425 (+ 226us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:47:05.160772 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:47:05.162010 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:47:05.163505 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:47:05.164362 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:47:05.164606 (+ 244us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:47:05.168361 (+ 3755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:47:05.168371 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:47:05.168385 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:47:05.168421 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:47:05.171589 (+ 3168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:47:05.172372 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:47:05.172379 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:47:05.172381 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:47:05.172534 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:47:05.172947 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:47:05.172951 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:47:05.172953 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:47:05.173618 (+ 665us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:47:05.173904 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0926 03:47:05.174524 (+ 620us) server_negotiation.cc:299] Negotiation successful
0926 03:47:05.174977 (+ 453us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":42,"server-negotiator.queue_time_us":1590,"spinlock_wait_cycles":54016,"thread_start_us":1371,"threads_started":1}
I20260926 03:47:05.180037 14284 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.194.253" port: 45537 } has no permanent_uuid. Determining permanent_uuid...
I20260926 03:47:05.181103 14286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:05.180698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.253:45537 (local address 127.0.0.1:34314)
0926 03:47:05.180877 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0926 03:47:05.180982 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.253:45537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260926 03:47:05.181383 14236 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.194.253:45537: connect: Connection refused (error 111)
I20260926 03:47:05.183049 14286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:05.182661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.253:45537 (local address 127.0.0.1:34322)
0926 03:47:05.182826 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0926 03:47:05.182907 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.253:45537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260926 03:47:05.183239 14284 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.194.253:45537: Network error: Client connection negotiation failed: client connection to 127.0.194.253:45537: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.194.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790394221564982-779-0/krb5kdc/kudu_127.0.194.253.keytab.
Entry for principal kudu/127.0.194.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790394221564982-779-0/krb5kdc/kudu_127.0.194.253.keytab.
I20260926 03:47:05.202548 779 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
/tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790394221564982-779-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790394221564982-779-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790394221564982-779-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790394221564982-779-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.194.253:45537
--webserver_interface=127.0.194.253
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790394221564982-779-0/krb5kdc/kudu_127.0.194.253.keytab
--principal=kudu/127.0.194.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.194.212:36299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.194.254:41069,127.0.194.253:45537,127.0.194.252:34233
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790394221564982-779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790394221564982-779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790394221564982-779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 03:47:05.232585 14284 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.194.253" port: 45537 } attempt: 1
I20260926 03:47:05.233777 14286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:05.233292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.253:45537 (local address 127.0.0.1:34336)
0926 03:47:05.233507 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0926 03:47:05.233641 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.253:45537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260926 03:47:05.234923 14286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:05.234498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.253:45537 (local address 127.0.0.1:34342)
0926 03:47:05.234654 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0926 03:47:05.234721 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.253:45537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260926 03:47:05.235106 14284 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.194.253:45537: Network error: Client connection negotiation failed: client connection to 127.0.194.253:45537: connect: Connection refused (error 111)
I20260926 03:47:05.303385 14284 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.194.253" port: 45537 } attempt: 2
I20260926 03:47:05.304593 14286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:05.304093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.253:45537 (local address 127.0.0.1:34354)
0926 03:47:05.304358 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0926 03:47:05.304453 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.253:45537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260926 03:47:05.305671 14286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:05.305290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.253:45537 (local address 127.0.0.1:34362)
0926 03:47:05.305467 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0926 03:47:05.305535 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.253:45537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260926 03:47:05.305850 14284 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.194.253:45537: Network error: Client connection negotiation failed: client connection to 127.0.194.253:45537: connect: Connection refused (error 111)
I20260926 03:47:05.397298 14284 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.194.253" port: 45537 } attempt: 3
I20260926 03:47:05.398505 14286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:05.398042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.253:45537 (local address 127.0.0.1:34368)
0926 03:47:05.398251 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0926 03:47:05.398375 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.253:45537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260926 03:47:05.399596 14286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:05.399230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.253:45537 (local address 127.0.0.1:34382)
0926 03:47:05.399400 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0926 03:47:05.399467 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.253:45537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260926 03:47:05.399768 14284 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.194.253:45537: Network error: Client connection negotiation failed: client connection to 127.0.194.253:45537: connect: Connection refused (error 111)
W20260926 03:47:05.535050 14291 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:47:05.535406 14291 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:47:05.535568 14291 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:47:05.542925 14291 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:47:05.543089 14291 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:47:05.543203 14291 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
I20260926 03:47:05.543028 14284 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.194.253" port: 45537 } attempt: 4
W20260926 03:47:05.543310 14291 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:47:05.543382 14291 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:47:05.544396 14286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:05.543816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.253:45537 (local address 127.0.0.1:34398)
0926 03:47:05.544152 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0926 03:47:05.544256 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.253:45537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149}
I20260926 03:47:05.545567 14286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:05.545142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.253:45537 (local address 127.0.0.1:34404)
0926 03:47:05.545317 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0926 03:47:05.545438 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.253:45537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260926 03:47:05.545769 14284 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.194.253:45537: Network error: Client connection negotiation failed: client connection to 127.0.194.253:45537: connect: Connection refused (error 111)
I20260926 03:47:05.554021 14291 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.194.212:36299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790394221564982-779-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790394221564982-779-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.194.254:41069,127.0.194.253:45537,127.0.194.252:34233
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790394221564982-779-0/krb5kdc/kudu_127.0.194.253.keytab
--principal=kudu/127.0.194.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.194.253:45537
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790394221564982-779-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.194.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790394221564982-779-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:47:05.555451 14291 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:47:05.556773 14291 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:47:05.568735 14296 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:47:05.568790 14297 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:47:05.568998 14299 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:47:05.570024 14291 server_base.cc:1034] running on GCE node
I20260926 03:47:05.571000 14291 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:47:05.572484 14291 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:47:05.573738 14291 hybrid_clock.cc:648] HybridClock initialized: now 1790394425573689 us; error 47 us; skew 500 ppm
I20260926 03:47:05.574126 14291 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:47:05 dist-test-slave-n2th krb5kdc[14204](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394425, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:47:05.577448 14291 init.cc:377] Logged in from keytab as kudu/127.0.194.253@KRBTEST.COM (short username kudu)
I20260926 03:47:05.579075 14291 webserver.cc:460] Webserver started at http://127.0.194.253:36667/ using document root <none> and password file <none>
I20260926 03:47:05.579795 14291 fs_manager.cc:359] Metadata directory not provided
I20260926 03:47:05.579910 14291 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:47:05.580165 14291 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:47:05.583413 14291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790394221564982-779-0/minicluster-data/master-1/data/instance:
uuid: "129d69f334524d43bba3515f98e5ab37"
format_stamp: "Formatted at 2026-09-26 03:47:05 on dist-test-slave-n2th"
server_key: "d677ab30aa26845993778376f8dc799b"
server_key_iv: "c0404b04dc2d6c696a46040c2ab5e042"
server_key_version: "encryptionkey@0"
I20260926 03:47:05.584223 14291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790394221564982-779-0/minicluster-data/master-1/wal/instance:
uuid: "129d69f334524d43bba3515f98e5ab37"
format_stamp: "Formatted at 2026-09-26 03:47:05 on dist-test-slave-n2th"
server_key: "d677ab30aa26845993778376f8dc799b"
server_key_iv: "c0404b04dc2d6c696a46040c2ab5e042"
server_key_version: "encryptionkey@0"
I20260926 03:47:05.589952 14291 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260926 03:47:05.593709 14306 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:47:05.595355 14291 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 03:47:05.595570 14291 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790394221564982-779-0/minicluster-data/master-1/data,/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790394221564982-779-0/minicluster-data/master-1/wal
uuid: "129d69f334524d43bba3515f98e5ab37"
format_stamp: "Formatted at 2026-09-26 03:47:05 on dist-test-slave-n2th"
server_key: "d677ab30aa26845993778376f8dc799b"
server_key_iv: "c0404b04dc2d6c696a46040c2ab5e042"
server_key_version: "encryptionkey@0"
I20260926 03:47:05.595767 14291 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790394221564982-779-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790394221564982-779-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790394221564982-779-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:47:05.614132 14291 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:47:05.619128 14291 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:47:05.619473 14291 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:47:05.638650 14291 rpc_server.cc:307] RPC server started. Bound to: 127.0.194.253:45537
I20260926 03:47:05.638674 14357 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.194.253:45537 every 8 connection(s)
I20260926 03:47:05.640187 14291 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790394221564982-779-0/minicluster-data/master-1/data/info.pb
I20260926 03:47:05.644251 14358 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:47:05.645916 779 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu as pid 14291
I20260926 03:47:05.646140 779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790394221564982-779-0/minicluster-data/master-1/wal/instance
I20260926 03:47:05.646772 779 external_mini_cluster.cc:1428] Setting key fc5d811a800cae73b95da95cd2f653b1
I20260926 03:47:05.654711 14358 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.194.254" port: 41069 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.194.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.194.252@KRBTEST.COM" created.
Sep 26 03:47:05 dist-test-slave-n2th krb5kdc[14204](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394425, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.253@KRBTEST.COM for kudu/127.0.194.254@KRBTEST.COM
I20260926 03:47:05.680178 14287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:05.657858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57994 (local address 127.0.194.254:41069)
0926 03:47:05.658098 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0926 03:47:05.658107 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:47:05.659433 (+ 1326us) server_negotiation.cc:407] Connection header received
0926 03:47:05.661357 (+ 1924us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:47:05.661375 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:47:05.661475 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:47:05.661602 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:47:05.663241 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:47:05.664576 (+ 1335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:47:05.666344 (+ 1768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:47:05.667242 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:47:05.667407 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:47:05.673604 (+ 6197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:47:05.673609 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:47:05.673612 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:47:05.673653 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:47:05.676682 (+ 3029us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:47:05.677611 (+ 929us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:47:05.677624 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:47:05.677627 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:47:05.677767 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:47:05.678250 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:47:05.678255 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:47:05.678264 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:47:05.678587 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:47:05.678783 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0926 03:47:05.679579 (+ 796us) server_negotiation.cc:299] Negotiation successful
0926 03:47:05.679839 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260926 03:47:05.681317 14361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:05.658347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:41069 (local address 127.0.0.1:57994)
0926 03:47:05.659244 (+ 897us) negotiation.cc:107] Waiting for socket to connect
0926 03:47:05.659297 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 03:47:05.661064 (+ 1767us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:47:05.661813 (+ 749us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:47:05.661822 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:47:05.662358 (+ 536us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:47:05.663042 (+ 684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:47:05.663062 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:47:05.664754 (+ 1692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:47:05.664757 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:47:05.666162 (+ 1405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:47:05.666180 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:47:05.667479 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:47:05.667485 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:47:05.667697 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:47:05.668477 (+ 780us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:47:05.668501 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:47:05.673297 (+ 4796us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:47:05.676896 (+ 3599us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:47:05.676907 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:47:05.676930 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:47:05.677452 (+ 522us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:47:05.677894 (+ 442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:47:05.677900 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:47:05.677904 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:47:05.678114 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:47:05.678770 (+ 656us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:47:05.678786 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:47:05.679180 (+ 394us) client_negotiation.cc:769] Sending connection context
0926 03:47:05.679465 (+ 285us) client_negotiation.cc:240] Negotiation successful
0926 03:47:05.680043 (+ 578us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":626,"thread_start_us":417,"threads_started":1}
I20260926 03:47:05.683168 14358 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.194.253" port: 45537 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 03:47:05 dist-test-slave-n2th krb5kdc[14204](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394425, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.253@KRBTEST.COM for kudu/127.0.194.253@KRBTEST.COM
I20260926 03:47:05.698369 14361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:05.683795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.253:45537 (local address 127.0.0.1:34412)
0926 03:47:05.683965 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0926 03:47:05.683986 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 03:47:05.684090 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:47:05.685176 (+ 1086us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:47:05.685180 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:47:05.685645 (+ 465us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:47:05.685773 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:47:05.685782 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:47:05.687185 (+ 1403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:47:05.687190 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:47:05.688016 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:47:05.688027 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:47:05.688880 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:47:05.688885 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:47:05.689078 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:47:05.689682 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:47:05.689700 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:47:05.692233 (+ 2533us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:47:05.695636 (+ 3403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:47:05.695640 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:47:05.695643 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:47:05.696256 (+ 613us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:47:05.696623 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:47:05.696627 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:47:05.696629 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:47:05.696776 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:47:05.697401 (+ 625us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:47:05.697404 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:47:05.697519 (+ 115us) client_negotiation.cc:769] Sending connection context
0926 03:47:05.697685 (+ 166us) client_negotiation.cc:240] Negotiation successful
0926 03:47:05.697983 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260926 03:47:05.698395 14363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:05.684068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34412 (local address 127.0.194.253:45537)
0926 03:47:05.684733 (+ 665us) server_negotiation.cc:206] Beginning negotiation
0926 03:47:05.684739 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:47:05.684764 (+ 25us) server_negotiation.cc:407] Connection header received
0926 03:47:05.684817 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:47:05.684822 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:47:05.685025 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:47:05.685186 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:47:05.685922 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:47:05.687059 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:47:05.688149 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:47:05.688703 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:47:05.688856 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:47:05.692457 (+ 3601us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:47:05.692470 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:47:05.692485 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:47:05.692514 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:47:05.695482 (+ 2968us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:47:05.696378 (+ 896us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:47:05.696385 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:47:05.696387 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:47:05.696501 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:47:05.696869 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:47:05.696872 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:47:05.696874 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:47:05.697287 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:47:05.697447 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0926 03:47:05.697839 (+ 392us) server_negotiation.cc:299] Negotiation successful
0926 03:47:05.698084 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":41,"server-negotiator.queue_time_us":503,"thread_start_us":343,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.194.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790394221564982-779-0/krb5kdc/kudu_127.0.194.252.keytab.
Entry for principal kudu/127.0.194.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790394221564982-779-0/krb5kdc/kudu_127.0.194.252.keytab.
I20260926 03:47:05.701138 14358 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.194.252" port: 34233 } has no permanent_uuid. Determining permanent_uuid...
I20260926 03:47:05.702399 14361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:05.701953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.252:34233 (local address 127.0.0.1:59632)
0926 03:47:05.702136 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0926 03:47:05.702248 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.252:34233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260926 03:47:05.702540 779 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
/tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790394221564982-779-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790394221564982-779-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790394221564982-779-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790394221564982-779-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.194.252:34233
--webserver_interface=127.0.194.252
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790394221564982-779-0/krb5kdc/kudu_127.0.194.252.keytab
--principal=kudu/127.0.194.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.194.212:36299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.194.254:41069,127.0.194.253:45537,127.0.194.252:34233
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790394221564982-779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790394221564982-779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790394221564982-779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:47:05.702848 14307 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.194.252:34233: connect: Connection refused (error 111)
I20260926 03:47:05.704427 14361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:05.704105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.252:34233 (local address 127.0.0.1:59640)
0926 03:47:05.704241 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0926 03:47:05.704323 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.252:34233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260926 03:47:05.704726 14358 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.194.252:34233: Network error: Client connection negotiation failed: client connection to 127.0.194.252:34233: connect: Connection refused (error 111)
I20260926 03:47:05.754040 14358 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.194.252" port: 34233 } attempt: 1
I20260926 03:47:05.755226 14361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:05.754720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.252:34233 (local address 127.0.0.1:59648)
0926 03:47:05.754963 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0926 03:47:05.755074 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.252:34233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260926 03:47:05.756315 14361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:05.755982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.252:34233 (local address 127.0.0.1:59654)
0926 03:47:05.756150 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0926 03:47:05.756211 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.252:34233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260926 03:47:05.756533 14358 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.194.252:34233: Network error: Client connection negotiation failed: client connection to 127.0.194.252:34233: connect: Connection refused (error 111)
I20260926 03:47:05.824757 14358 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.194.252" port: 34233 } attempt: 2
I20260926 03:47:05.825938 14361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:05.825478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.252:34233 (local address 127.0.0.1:59670)
0926 03:47:05.825673 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0926 03:47:05.825776 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.252:34233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260926 03:47:05.827107 14361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:05.826712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.252:34233 (local address 127.0.0.1:59672)
0926 03:47:05.826919 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0926 03:47:05.826991 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.252:34233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114}
W20260926 03:47:05.827417 14358 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.194.252:34233: Network error: Client connection negotiation failed: client connection to 127.0.194.252:34233: connect: Connection refused (error 111)
I20260926 03:47:05.845043 14284 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.194.253" port: 45537 } attempt: 5
Sep 26 03:47:05 dist-test-slave-n2th krb5kdc[14204](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394425, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.254@KRBTEST.COM for kudu/127.0.194.253@KRBTEST.COM
I20260926 03:47:05.863739 14363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:05.845941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34416 (local address 127.0.194.253:45537)
0926 03:47:05.846131 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0926 03:47:05.846137 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:47:05.846159 (+ 22us) server_negotiation.cc:407] Connection header received
0926 03:47:05.846395 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:47:05.846399 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:47:05.846464 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:47:05.846572 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:47:05.847476 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:47:05.848186 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:47:05.849413 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:47:05.849840 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:47:05.849976 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:47:05.857693 (+ 7717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:47:05.857699 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:47:05.857702 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:47:05.857741 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:47:05.860738 (+ 2997us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:47:05.861560 (+ 822us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:47:05.861573 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:47:05.861576 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:47:05.861723 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:47:05.862160 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:47:05.862166 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:47:05.862169 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:47:05.862501 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:47:05.862676 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0926 03:47:05.863153 (+ 477us) server_negotiation.cc:299] Negotiation successful
0926 03:47:05.863387 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 03:47:05.863739 14286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:05.845821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.253:45537 (local address 127.0.0.1:34416)
0926 03:47:05.846040 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0926 03:47:05.846064 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 03:47:05.846200 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:47:05.846605 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:47:05.846610 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:47:05.847099 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:47:05.847315 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:47:05.847324 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:47:05.848326 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:47:05.848334 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:47:05.849263 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:47:05.849276 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:47:05.850014 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:47:05.850019 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:47:05.850208 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:47:05.854958 (+ 4750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:47:05.854987 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:47:05.857420 (+ 2433us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:47:05.860947 (+ 3527us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:47:05.860954 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:47:05.860959 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:47:05.861405 (+ 446us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:47:05.861866 (+ 461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:47:05.861871 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:47:05.861875 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:47:05.862034 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:47:05.862632 (+ 598us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:47:05.862637 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:47:05.862886 (+ 249us) client_negotiation.cc:769] Sending connection context
0926 03:47:05.863050 (+ 164us) client_negotiation.cc:240] Negotiation successful
0926 03:47:05.863375 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69}
I20260926 03:47:05.864971 14284 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.194.252" port: 34233 } has no permanent_uuid. Determining permanent_uuid...
I20260926 03:47:05.866267 14286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:05.865859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.252:34233 (local address 127.0.0.1:59682)
0926 03:47:05.866040 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0926 03:47:05.866122 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.252:34233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260926 03:47:05.867316 14286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:05.866957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.252:34233 (local address 127.0.0.1:59684)
0926 03:47:05.867129 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0926 03:47:05.867201 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.252:34233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260926 03:47:05.867520 14284 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.194.252:34233: Network error: Client connection negotiation failed: client connection to 127.0.194.252:34233: connect: Connection refused (error 111)
I20260926 03:47:05.918840 14358 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.194.252" port: 34233 } attempt: 3
I20260926 03:47:05.918831 14284 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.194.252" port: 34233 } attempt: 1
I20260926 03:47:05.920147 14361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:05.919643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.252:34233 (local address 127.0.0.1:59708)
0926 03:47:05.919870 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0926 03:47:05.919985 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.252:34233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260926 03:47:05.920298 14286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:05.919633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.252:34233 (local address 127.0.0.1:59694)
0926 03:47:05.919963 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0926 03:47:05.920086 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.252:34233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211}
I20260926 03:47:05.921411 14361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:05.920955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.252:34233 (local address 127.0.0.1:59710)
0926 03:47:05.921204 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0926 03:47:05.921287 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.252:34233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133}
W20260926 03:47:05.921793 14358 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.194.252:34233: Network error: Client connection negotiation failed: client connection to 127.0.194.252:34233: connect: Connection refused (error 111)
I20260926 03:47:05.921989 14286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:05.920988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.252:34233 (local address 127.0.0.1:59714)
0926 03:47:05.921720 (+ 732us) negotiation.cc:107] Waiting for socket to connect
0926 03:47:05.921829 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.252:34233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260926 03:47:05.922214 14284 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.194.252:34233: Network error: Client connection negotiation failed: client connection to 127.0.194.252:34233: connect: Connection refused (error 111)
I20260926 03:47:05.990476 14284 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.194.252" port: 34233 } attempt: 2
I20260926 03:47:05.991751 14286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:05.991234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.252:34233 (local address 127.0.0.1:59728)
0926 03:47:05.991453 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0926 03:47:05.991567 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.252:34233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260926 03:47:05.992863 14286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:05.992495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.252:34233 (local address 127.0.0.1:59738)
0926 03:47:05.992655 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0926 03:47:05.992743 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.252:34233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260926 03:47:05.993249 14284 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.194.252:34233: Network error: Client connection negotiation failed: client connection to 127.0.194.252:34233: connect: Connection refused (error 111)
I20260926 03:47:06.065011 14358 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.194.252" port: 34233 } attempt: 4
I20260926 03:47:06.066402 14361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:06.065758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.252:34233 (local address 127.0.0.1:59750)
0926 03:47:06.066033 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0926 03:47:06.066163 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.252:34233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260926 03:47:06.067538 14361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:06.067102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.252:34233 (local address 127.0.0.1:59764)
0926 03:47:06.067265 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0926 03:47:06.067380 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.252:34233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260926 03:47:06.067785 14358 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.194.252:34233: Network error: Client connection negotiation failed: client connection to 127.0.194.252:34233: connect: Connection refused (error 111)
W20260926 03:47:06.073664 14365 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:47:06.073968 14365 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:47:06.074132 14365 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:47:06.081879 14365 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:47:06.082002 14365 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:47:06.082099 14365 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:47:06.082201 14365 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:47:06.082269 14365 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:47:06.092948 14365 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.194.212:36299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790394221564982-779-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790394221564982-779-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.194.254:41069,127.0.194.253:45537,127.0.194.252:34233
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790394221564982-779-0/krb5kdc/kudu_127.0.194.252.keytab
--principal=kudu/127.0.194.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.194.252:34233
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790394221564982-779-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.194.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790394221564982-779-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:47:06.094331 14365 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:47:06.095937 14365 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 03:47:06.099548 14284 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.194.252" port: 34233 } attempt: 3
I20260926 03:47:06.100847 14286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:06.100321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.252:34233 (local address 127.0.0.1:59778)
0926 03:47:06.100591 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0926 03:47:06.100678 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.252:34233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260926 03:47:06.102049 14286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:06.101615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.252:34233 (local address 127.0.0.1:59794)
0926 03:47:06.101824 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0926 03:47:06.101911 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.252:34233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
W20260926 03:47:06.102370 14284 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.194.252:34233: Network error: Client connection negotiation failed: client connection to 127.0.194.252:34233: connect: Connection refused (error 111)
W20260926 03:47:06.108840 14370 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:47:06.108917 14373 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:47:06.108886 14371 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:47:06.110478 14365 server_base.cc:1034] running on GCE node
I20260926 03:47:06.111368 14365 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:47:06.112880 14365 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:47:06.114130 14365 hybrid_clock.cc:648] HybridClock initialized: now 1790394426114071 us; error 59 us; skew 500 ppm
I20260926 03:47:06.114506 14365 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:47:06 dist-test-slave-n2th krb5kdc[14204](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394426, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:47:06.117774 14365 init.cc:377] Logged in from keytab as kudu/127.0.194.252@KRBTEST.COM (short username kudu)
I20260926 03:47:06.119472 14365 webserver.cc:460] Webserver started at http://127.0.194.252:38499/ using document root <none> and password file <none>
I20260926 03:47:06.120219 14365 fs_manager.cc:359] Metadata directory not provided
I20260926 03:47:06.120376 14365 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:47:06.120692 14365 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:47:06.123898 14365 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790394221564982-779-0/minicluster-data/master-2/data/instance:
uuid: "64062209971045e0b7737c1e8d356abb"
format_stamp: "Formatted at 2026-09-26 03:47:06 on dist-test-slave-n2th"
server_key: "3d50b2aac59add4bb649e9028d215aa9"
server_key_iv: "119c767827c22809ce1926041fd098ca"
server_key_version: "encryptionkey@0"
I20260926 03:47:06.124756 14365 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790394221564982-779-0/minicluster-data/master-2/wal/instance:
uuid: "64062209971045e0b7737c1e8d356abb"
format_stamp: "Formatted at 2026-09-26 03:47:06 on dist-test-slave-n2th"
server_key: "3d50b2aac59add4bb649e9028d215aa9"
server_key_iv: "119c767827c22809ce1926041fd098ca"
server_key_version: "encryptionkey@0"
I20260926 03:47:06.130276 14365 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 03:47:06.134045 14380 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:47:06.135486 14365 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 03:47:06.135681 14365 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790394221564982-779-0/minicluster-data/master-2/data,/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790394221564982-779-0/minicluster-data/master-2/wal
uuid: "64062209971045e0b7737c1e8d356abb"
format_stamp: "Formatted at 2026-09-26 03:47:06 on dist-test-slave-n2th"
server_key: "3d50b2aac59add4bb649e9028d215aa9"
server_key_iv: "119c767827c22809ce1926041fd098ca"
server_key_version: "encryptionkey@0"
I20260926 03:47:06.135908 14365 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790394221564982-779-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790394221564982-779-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790394221564982-779-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:47:06.151026 14365 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:47:06.155138 14365 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:47:06.155455 14365 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:47:06.174088 14365 rpc_server.cc:307] RPC server started. Bound to: 127.0.194.252:34233
I20260926 03:47:06.174118 14431 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.194.252:34233 every 8 connection(s)
I20260926 03:47:06.175614 14365 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790394221564982-779-0/minicluster-data/master-2/data/info.pb
I20260926 03:47:06.179781 14432 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:47:06.179914 779 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu as pid 14365
I20260926 03:47:06.180114 779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790394221564982-779-0/minicluster-data/master-2/wal/instance
I20260926 03:47:06.180675 779 external_mini_cluster.cc:1428] Setting key 177a9880efb0f7619c63c328a70b7083
I20260926 03:47:06.189103 14432 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.194.254" port: 41069 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 03:47:06 dist-test-slave-n2th krb5kdc[14204](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394424, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.194.252@KRBTEST.COM
Sep 26 03:47:06 dist-test-slave-n2th krb5kdc[14204](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394426, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.252@KRBTEST.COM for kudu/127.0.194.254@KRBTEST.COM
I20260926 03:47:06.204169 14435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:06.182607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59802 (local address 127.0.194.252:34233)
0926 03:47:06.183534 (+ 927us) server_negotiation.cc:206] Beginning negotiation
0926 03:47:06.183546 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 03:47:06.183585 (+ 39us) server_negotiation.cc:407] Connection header received
0926 03:47:06.185045 (+ 1460us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:47:06.185066 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:47:06.185500 (+ 434us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:47:06.185951 (+ 451us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:47:06.187525 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:47:06.188798 (+ 1273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:47:06.190240 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:47:06.190998 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:47:06.191183 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:47:06.195773 (+ 4590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:47:06.195783 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:47:06.195796 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:47:06.195848 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:47:06.199474 (+ 3626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:47:06.200387 (+ 913us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:47:06.200394 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:47:06.200401 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:47:06.200611 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:47:06.201056 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:47:06.201061 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:47:06.201065 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:47:06.201708 (+ 643us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:47:06.202007 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0926 03:47:06.202416 (+ 409us) server_negotiation.cc:299] Negotiation successful
0926 03:47:06.202972 (+ 556us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":723,"thread_start_us":431,"threads_started":1}
I20260926 03:47:06.209376 14437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:06.191841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58006 (local address 127.0.194.254:41069)
0926 03:47:06.194455 (+ 2614us) server_negotiation.cc:206] Beginning negotiation
0926 03:47:06.194459 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:47:06.194474 (+ 15us) server_negotiation.cc:407] Connection header received
0926 03:47:06.194523 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:47:06.194527 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:47:06.194581 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:47:06.194707 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:47:06.195966 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:47:06.196761 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:47:06.198895 (+ 2134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:47:06.199331 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:47:06.199433 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:47:06.203197 (+ 3764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:47:06.203202 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:47:06.203205 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:47:06.203238 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:47:06.206337 (+ 3099us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:47:06.207213 (+ 876us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:47:06.207223 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:47:06.207226 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:47:06.207398 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:47:06.207897 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:47:06.207902 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:47:06.207905 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:47:06.208280 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:47:06.208444 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0926 03:47:06.208932 (+ 488us) server_negotiation.cc:299] Negotiation successful
0926 03:47:06.209093 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2508,"thread_start_us":2369,"threads_started":1}
I20260926 03:47:06.209641 14436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:06.191765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:41069 (local address 127.0.0.1:58006)
0926 03:47:06.192591 (+ 826us) negotiation.cc:107] Waiting for socket to connect
0926 03:47:06.192639 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 03:47:06.192782 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:47:06.194801 (+ 2019us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:47:06.194809 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:47:06.195416 (+ 607us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:47:06.195784 (+ 368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:47:06.195795 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:47:06.196930 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:47:06.196936 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:47:06.198682 (+ 1746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:47:06.198697 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:47:06.199524 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:47:06.199532 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:47:06.199722 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:47:06.200364 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:47:06.200390 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:47:06.203010 (+ 2620us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:47:06.206510 (+ 3500us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:47:06.206523 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:47:06.206532 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:47:06.207075 (+ 543us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:47:06.207543 (+ 468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:47:06.207550 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:47:06.207556 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:47:06.207746 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:47:06.208400 (+ 654us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:47:06.208408 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:47:06.208616 (+ 208us) client_negotiation.cc:769] Sending connection context
0926 03:47:06.208823 (+ 207us) client_negotiation.cc:240] Negotiation successful
0926 03:47:06.209179 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":555,"thread_start_us":391,"threads_started":1}
I20260926 03:47:06.211180 14432 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.194.253" port: 45537 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 03:47:06 dist-test-slave-n2th krb5kdc[14204](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394424, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.194.254@KRBTEST.COM
Sep 26 03:47:06 dist-test-slave-n2th krb5kdc[14204](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394426, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.252@KRBTEST.COM for kudu/127.0.194.253@KRBTEST.COM
I20260926 03:47:06.223475 14438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:06.207409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58008 (local address 127.0.194.254:41069)
0926 03:47:06.208054 (+ 645us) server_negotiation.cc:206] Beginning negotiation
0926 03:47:06.208059 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:47:06.208081 (+ 22us) server_negotiation.cc:407] Connection header received
0926 03:47:06.208122 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:47:06.208126 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:47:06.208187 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:47:06.208296 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:47:06.210121 (+ 1825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:47:06.211261 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:47:06.212732 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:47:06.213148 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:47:06.213296 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:47:06.217588 (+ 4292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:47:06.217594 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:47:06.217597 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:47:06.217630 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:47:06.220616 (+ 2986us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:47:06.221310 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:47:06.221318 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:47:06.221325 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:47:06.221453 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:47:06.221858 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:47:06.221862 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:47:06.221865 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:47:06.222257 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:47:06.222515 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0926 03:47:06.222847 (+ 332us) server_negotiation.cc:299] Negotiation successful
0926 03:47:06.223079 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":486,"thread_start_us":265,"threads_started":1}
I20260926 03:47:06.223917 14436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:06.211797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.253:45537 (local address 127.0.0.1:34426)
0926 03:47:06.212160 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0926 03:47:06.212179 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 03:47:06.212288 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:47:06.212607 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:47:06.212611 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:47:06.212968 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:47:06.213103 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:47:06.213110 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:47:06.214030 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:47:06.214038 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:47:06.214978 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:47:06.214989 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:47:06.215686 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:47:06.215691 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:47:06.215905 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:47:06.216442 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:47:06.216465 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:47:06.218388 (+ 1923us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:47:06.221487 (+ 3099us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:47:06.221492 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:47:06.221496 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:47:06.221856 (+ 360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:47:06.222292 (+ 436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:47:06.222295 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:47:06.222298 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:47:06.222431 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:47:06.222998 (+ 567us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:47:06.223002 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:47:06.223145 (+ 143us) client_negotiation.cc:769] Sending connection context
0926 03:47:06.223343 (+ 198us) client_negotiation.cc:240] Negotiation successful
0926 03:47:06.223628 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264}
I20260926 03:47:06.224279 14363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:06.211973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34426 (local address 127.0.194.253:45537)
0926 03:47:06.212144 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0926 03:47:06.212150 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:47:06.212226 (+ 76us) server_negotiation.cc:407] Connection header received
0926 03:47:06.212422 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:47:06.212426 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:47:06.212489 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:47:06.212588 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:47:06.213246 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:47:06.213898 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:47:06.215111 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:47:06.215534 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:47:06.215658 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:47:06.218583 (+ 2925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:47:06.218588 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:47:06.218591 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:47:06.218623 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:47:06.221347 (+ 2724us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:47:06.222013 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:47:06.222019 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:47:06.222021 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:47:06.222167 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:47:06.222547 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:47:06.222552 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:47:06.222557 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:47:06.222883 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:47:06.223039 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0926 03:47:06.223476 (+ 437us) server_negotiation.cc:299] Negotiation successful
0926 03:47:06.223842 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 03:47:06.225335 14432 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.194.252" port: 34233 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 03:47:06 dist-test-slave-n2th krb5kdc[14204](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394426, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.252@KRBTEST.COM for kudu/127.0.194.252@KRBTEST.COM
Sep 26 03:47:06 dist-test-slave-n2th krb5kdc[14204](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394424, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.194.253@KRBTEST.COM
I20260926 03:47:06.239343 14436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:06.226146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.252:34233 (local address 127.0.0.1:59812)
0926 03:47:06.226297 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0926 03:47:06.226315 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 03:47:06.226404 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:47:06.226684 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:47:06.226686 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:47:06.227098 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:47:06.227226 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:47:06.227234 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:47:06.228178 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:47:06.228183 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:47:06.229072 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:47:06.229082 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:47:06.229924 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:47:06.229930 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:47:06.230133 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:47:06.230810 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:47:06.230834 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:47:06.233081 (+ 2247us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:47:06.236780 (+ 3699us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:47:06.236784 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:47:06.236787 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:47:06.237155 (+ 368us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:47:06.237561 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:47:06.237564 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:47:06.237566 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:47:06.237736 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:47:06.238287 (+ 551us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:47:06.238294 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:47:06.238457 (+ 163us) client_negotiation.cc:769] Sending connection context
0926 03:47:06.238616 (+ 159us) client_negotiation.cc:240] Negotiation successful
0926 03:47:06.238973 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260926 03:47:06.239704 14435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:06.226235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59812 (local address 127.0.194.252:34233)
0926 03:47:06.226403 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0926 03:47:06.226409 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:47:06.226426 (+ 17us) server_negotiation.cc:407] Connection header received
0926 03:47:06.226479 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:47:06.226486 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:47:06.226554 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:47:06.226684 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:47:06.227369 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:47:06.228051 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:47:06.229208 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:47:06.229761 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:47:06.229856 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:47:06.233278 (+ 3422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:47:06.233284 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:47:06.233287 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:47:06.233316 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:47:06.236581 (+ 3265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:47:06.237286 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:47:06.237291 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:47:06.237295 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:47:06.237434 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:47:06.237859 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:47:06.237866 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:47:06.237871 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:47:06.238163 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:47:06.238270 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0926 03:47:06.238773 (+ 503us) server_negotiation.cc:299] Negotiation successful
0926 03:47:06.239179 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":90,"server-negotiator.queue_time_us":65}
I20260926 03:47:06.240803 14363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:06.225439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34432 (local address 127.0.194.253:45537)
0926 03:47:06.225610 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0926 03:47:06.225616 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:47:06.225639 (+ 23us) server_negotiation.cc:407] Connection header received
0926 03:47:06.225687 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:47:06.225691 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:47:06.225760 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:47:06.225887 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:47:06.227434 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:47:06.228163 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:47:06.229498 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:47:06.229978 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:47:06.230116 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:47:06.234479 (+ 4363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:47:06.234487 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:47:06.234492 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:47:06.234536 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:47:06.237732 (+ 3196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:47:06.238504 (+ 772us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:47:06.238508 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:47:06.238510 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:47:06.238687 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:47:06.239130 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:47:06.239135 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:47:06.239139 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:47:06.239514 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:47:06.239798 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0926 03:47:06.240325 (+ 527us) server_negotiation.cc:299] Negotiation successful
0926 03:47:06.240519 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 03:47:06.246140 14432 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 64062209971045e0b7737c1e8d356abb: Bootstrap starting.
I20260926 03:47:06.246953 779 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 03:47:06.250705 14432 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 64062209971045e0b7737c1e8d356abb: Neither blocks nor log segments found. Creating new log.
I20260926 03:47:06.252260 14432 log.cc:824] T 00000000000000000000000000000000 P 64062209971045e0b7737c1e8d356abb: Log is configured to *not* fsync() on all Append() calls
I20260926 03:47:06.256201 14432 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 64062209971045e0b7737c1e8d356abb: No bootstrap required, opened a new log
I20260926 03:47:06.268285 14432 raft_consensus.cc:348] T 00000000000000000000000000000000 P 64062209971045e0b7737c1e8d356abb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a42cb64b376e49b6af88f02210fb9d2d" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 41069 } } peers { permanent_uuid: "129d69f334524d43bba3515f98e5ab37" member_type: VOTER last_known_addr { host: "127.0.194.253" port: 45537 } } peers { permanent_uuid: "64062209971045e0b7737c1e8d356abb" member_type: VOTER last_known_addr { host: "127.0.194.252" port: 34233 } }
I20260926 03:47:06.268800 14432 raft_consensus.cc:374] T 00000000000000000000000000000000 P 64062209971045e0b7737c1e8d356abb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:47:06.268929 14432 raft_consensus.cc:729] T 00000000000000000000000000000000 P 64062209971045e0b7737c1e8d356abb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64062209971045e0b7737c1e8d356abb, State: Initialized, Role: FOLLOWER
I20260926 03:47:06.269675 14432 consensus_queue.cc:260] T 00000000000000000000000000000000 P 64062209971045e0b7737c1e8d356abb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a42cb64b376e49b6af88f02210fb9d2d" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 41069 } } peers { permanent_uuid: "129d69f334524d43bba3515f98e5ab37" member_type: VOTER last_known_addr { host: "127.0.194.253" port: 45537 } } peers { permanent_uuid: "64062209971045e0b7737c1e8d356abb" member_type: VOTER last_known_addr { host: "127.0.194.252" port: 34233 } }
I20260926 03:47:06.270838 14447 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64062209971045e0b7737c1e8d356abb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a42cb64b376e49b6af88f02210fb9d2d" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 41069 } } peers { permanent_uuid: "129d69f334524d43bba3515f98e5ab37" member_type: VOTER last_known_addr { host: "127.0.194.253" port: 45537 } } peers { permanent_uuid: "64062209971045e0b7737c1e8d356abb" member_type: VOTER last_known_addr { host: "127.0.194.252" port: 34233 } } }
I20260926 03:47:06.271235 14447 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64062209971045e0b7737c1e8d356abb [sys.catalog]: This master's current role is: FOLLOWER
I20260926 03:47:06.272347 14432 sys_catalog.cc:564] T 00000000000000000000000000000000 P 64062209971045e0b7737c1e8d356abb [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:47:06.272598 14435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:06.253114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59824 (local address 127.0.194.252:34233)
0926 03:47:06.253355 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0926 03:47:06.253362 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:47:06.254009 (+ 647us) server_negotiation.cc:407] Connection header received
0926 03:47:06.254257 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:47:06.254262 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:47:06.254339 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:47:06.254413 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:47:06.256931 (+ 2518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:47:06.258003 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:47:06.260186 (+ 2183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:47:06.260890 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:47:06.261001 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:47:06.265273 (+ 4272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:47:06.265280 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:47:06.265285 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:47:06.265331 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:47:06.269108 (+ 3777us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:47:06.269921 (+ 813us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:47:06.269927 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:47:06.269931 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:47:06.270101 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:47:06.270575 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:47:06.270582 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:47:06.270585 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:47:06.271049 (+ 464us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:47:06.271325 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0926 03:47:06.271923 (+ 598us) server_negotiation.cc:299] Negotiation successful
0926 03:47:06.272143 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 03:47:06.276968 14438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:06.253055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58016 (local address 127.0.194.254:41069)
0926 03:47:06.253277 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0926 03:47:06.253284 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:47:06.254592 (+ 1308us) server_negotiation.cc:407] Connection header received
0926 03:47:06.254916 (+ 324us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:47:06.254922 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:47:06.255003 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:47:06.255085 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:47:06.257913 (+ 2828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:47:06.259321 (+ 1408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:47:06.262242 (+ 2921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:47:06.263182 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:47:06.263321 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:47:06.269195 (+ 5874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:47:06.269203 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:47:06.269210 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:47:06.269256 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:47:06.273338 (+ 4082us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:47:06.274515 (+ 1177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:47:06.274522 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:47:06.274528 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:47:06.274682 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:47:06.275142 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:47:06.275160 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:47:06.275166 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:47:06.275643 (+ 477us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:47:06.275968 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0926 03:47:06.276369 (+ 401us) server_negotiation.cc:299] Negotiation successful
0926 03:47:06.276612 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 03:47:06.279803 14284 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.194.252" port: 34233 } attempt: 4
I20260926 03:47:06.289592 14363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:06.254701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34446 (local address 127.0.194.253:45537)
0926 03:47:06.254930 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0926 03:47:06.254936 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:47:06.272450 (+ 17514us) server_negotiation.cc:407] Connection header received
0926 03:47:06.272761 (+ 311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:47:06.272767 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:47:06.272864 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:47:06.272940 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:47:06.274654 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:47:06.275597 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:47:06.277512 (+ 1915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:47:06.278473 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:47:06.278644 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:47:06.282616 (+ 3972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:47:06.282626 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:47:06.282634 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:47:06.282684 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:47:06.286163 (+ 3479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:47:06.287179 (+ 1016us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:47:06.287184 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:47:06.287187 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:47:06.287351 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:47:06.287783 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:47:06.287791 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:47:06.287795 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:47:06.288299 (+ 504us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:47:06.288633 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0926 03:47:06.288948 (+ 315us) server_negotiation.cc:299] Negotiation successful
0926 03:47:06.289214 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
Sep 26 03:47:06 dist-test-slave-n2th krb5kdc[14204](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394425, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.254@KRBTEST.COM for kudu/127.0.194.252@KRBTEST.COM
I20260926 03:47:06.288189 14365 master_runner.cc:186] Error getting master registration for 127.0.194.254:41069: OK, instance_id { permanent_uuid: "a42cb64b376e49b6af88f02210fb9d2d" instance_seqno: 1790394425134131 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 03:47:06.299328 14365 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 03:47:06.299798 14435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:06.281063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59834 (local address 127.0.194.252:34233)
0926 03:47:06.281236 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0926 03:47:06.281241 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:47:06.281255 (+ 14us) server_negotiation.cc:407] Connection header received
0926 03:47:06.281305 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:47:06.281308 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:47:06.281382 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:47:06.281495 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:47:06.282194 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:47:06.284014 (+ 1820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:47:06.285957 (+ 1943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:47:06.286771 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:47:06.286899 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:47:06.292462 (+ 5563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:47:06.292469 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:47:06.292476 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:47:06.292524 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:47:06.296580 (+ 4056us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:47:06.297549 (+ 969us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:47:06.297554 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:47:06.297558 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:47:06.297704 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:47:06.298165 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:47:06.298170 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:47:06.298174 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:47:06.298596 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:47:06.298783 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0926 03:47:06.299254 (+ 471us) server_negotiation.cc:299] Negotiation successful
0926 03:47:06.299498 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 03:47:06.299944 14286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:06.280350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.252:34233 (local address 127.0.0.1:59834)
0926 03:47:06.280725 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0926 03:47:06.280741 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 03:47:06.280828 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:47:06.281518 (+ 690us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:47:06.281523 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:47:06.281906 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:47:06.282048 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:47:06.282057 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:47:06.284237 (+ 2180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:47:06.284245 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:47:06.285791 (+ 1546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:47:06.285806 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:47:06.287103 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:47:06.287113 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:47:06.287355 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:47:06.288813 (+ 1458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:47:06.288846 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:47:06.292215 (+ 3369us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:47:06.296785 (+ 4570us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:47:06.296791 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:47:06.296795 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:47:06.297405 (+ 610us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:47:06.297854 (+ 449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:47:06.297859 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:47:06.297863 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:47:06.298039 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:47:06.298776 (+ 737us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:47:06.298781 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:47:06.298976 (+ 195us) client_negotiation.cc:769] Sending connection context
0926 03:47:06.299142 (+ 166us) client_negotiation.cc:240] Negotiation successful
0926 03:47:06.299512 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251}
W20260926 03:47:06.300983 14458 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 64062209971045e0b7737c1e8d356abb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 03:47:06.301077 14458 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 03:47:06.305308 14284 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a42cb64b376e49b6af88f02210fb9d2d: Bootstrap starting.
I20260926 03:47:06.308530 14284 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a42cb64b376e49b6af88f02210fb9d2d: Neither blocks nor log segments found. Creating new log.
I20260926 03:47:06.310012 14284 log.cc:824] T 00000000000000000000000000000000 P a42cb64b376e49b6af88f02210fb9d2d: Log is configured to *not* fsync() on all Append() calls
I20260926 03:47:06.312889 14284 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a42cb64b376e49b6af88f02210fb9d2d: No bootstrap required, opened a new log
I20260926 03:47:06.322276 14284 raft_consensus.cc:348] T 00000000000000000000000000000000 P a42cb64b376e49b6af88f02210fb9d2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a42cb64b376e49b6af88f02210fb9d2d" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 41069 } } peers { permanent_uuid: "129d69f334524d43bba3515f98e5ab37" member_type: VOTER last_known_addr { host: "127.0.194.253" port: 45537 } } peers { permanent_uuid: "64062209971045e0b7737c1e8d356abb" member_type: VOTER last_known_addr { host: "127.0.194.252" port: 34233 } }
I20260926 03:47:06.322765 14284 raft_consensus.cc:374] T 00000000000000000000000000000000 P a42cb64b376e49b6af88f02210fb9d2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:47:06.322896 14284 raft_consensus.cc:729] T 00000000000000000000000000000000 P a42cb64b376e49b6af88f02210fb9d2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a42cb64b376e49b6af88f02210fb9d2d, State: Initialized, Role: FOLLOWER
I20260926 03:47:06.323612 14284 consensus_queue.cc:260] T 00000000000000000000000000000000 P a42cb64b376e49b6af88f02210fb9d2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a42cb64b376e49b6af88f02210fb9d2d" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 41069 } } peers { permanent_uuid: "129d69f334524d43bba3515f98e5ab37" member_type: VOTER last_known_addr { host: "127.0.194.253" port: 45537 } } peers { permanent_uuid: "64062209971045e0b7737c1e8d356abb" member_type: VOTER last_known_addr { host: "127.0.194.252" port: 34233 } }
I20260926 03:47:06.325726 14461 sys_catalog.cc:455] T 00000000000000000000000000000000 P a42cb64b376e49b6af88f02210fb9d2d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a42cb64b376e49b6af88f02210fb9d2d" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 41069 } } peers { permanent_uuid: "129d69f334524d43bba3515f98e5ab37" member_type: VOTER last_known_addr { host: "127.0.194.253" port: 45537 } } peers { permanent_uuid: "64062209971045e0b7737c1e8d356abb" member_type: VOTER last_known_addr { host: "127.0.194.252" port: 34233 } } }
I20260926 03:47:06.326072 14461 sys_catalog.cc:458] T 00000000000000000000000000000000 P a42cb64b376e49b6af88f02210fb9d2d [sys.catalog]: This master's current role is: FOLLOWER
I20260926 03:47:06.327265 14284 sys_catalog.cc:564] T 00000000000000000000000000000000 P a42cb64b376e49b6af88f02210fb9d2d [sys.catalog]: configured and running, proceeding with master startup.
W20260926 03:47:06.339278 14472 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a42cb64b376e49b6af88f02210fb9d2d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 03:47:06.339531 14472 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 03:47:06.336706 14217 master_runner.cc:186] Error getting master registration for 127.0.194.253:45537: OK, instance_id { permanent_uuid: "129d69f334524d43bba3515f98e5ab37" instance_seqno: 1790394425633304 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 03:47:06.350236 14217 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 03:47:06.367017 14358 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.194.252" port: 34233 } attempt: 5
W20260926 03:47:06.373409 14238 tablet.cc:2369] T 00000000000000000000000000000000 P a42cb64b376e49b6af88f02210fb9d2d: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 26 03:47:06 dist-test-slave-n2th krb5kdc[14204](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394425, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.253@KRBTEST.COM for kudu/127.0.194.252@KRBTEST.COM
I20260926 03:47:06.386047 14361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:06.367720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.252:34233 (local address 127.0.0.1:59846)
0926 03:47:06.367998 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0926 03:47:06.368022 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 03:47:06.368160 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:47:06.368560 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:47:06.368565 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:47:06.369108 (+ 543us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:47:06.369336 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:47:06.369351 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:47:06.370856 (+ 1505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:47:06.370862 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:47:06.372336 (+ 1474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:47:06.372353 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:47:06.373762 (+ 1409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:47:06.373768 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:47:06.373995 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:47:06.374761 (+ 766us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:47:06.374788 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:47:06.377906 (+ 3118us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:47:06.382790 (+ 4884us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:47:06.382797 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:47:06.382802 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:47:06.383408 (+ 606us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:47:06.383941 (+ 533us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:47:06.383946 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:47:06.383950 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:47:06.384129 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:47:06.384814 (+ 685us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:47:06.384820 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:47:06.384992 (+ 172us) client_negotiation.cc:769] Sending connection context
0926 03:47:06.385165 (+ 173us) client_negotiation.cc:240] Negotiation successful
0926 03:47:06.385498 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156}
I20260926 03:47:06.386077 14435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:06.367816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59846 (local address 127.0.194.252:34233)
0926 03:47:06.367999 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0926 03:47:06.368005 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:47:06.368109 (+ 104us) server_negotiation.cc:407] Connection header received
0926 03:47:06.368306 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:47:06.368312 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:47:06.368391 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:47:06.368498 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:47:06.369511 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:47:06.370680 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:47:06.372530 (+ 1850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:47:06.373271 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:47:06.373438 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:47:06.378140 (+ 4702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:47:06.378147 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:47:06.378152 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:47:06.378203 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:47:06.382557 (+ 4354us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:47:06.383577 (+ 1020us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:47:06.383584 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:47:06.383588 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:47:06.383799 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:47:06.384260 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:47:06.384266 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:47:06.384269 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:47:06.384654 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:47:06.384830 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0926 03:47:06.385303 (+ 473us) server_negotiation.cc:299] Negotiation successful
0926 03:47:06.385653 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 03:47:06.392794 14358 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 129d69f334524d43bba3515f98e5ab37: Bootstrap starting.
I20260926 03:47:06.397254 14358 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 129d69f334524d43bba3515f98e5ab37: Neither blocks nor log segments found. Creating new log.
I20260926 03:47:06.398703 14358 log.cc:824] T 00000000000000000000000000000000 P 129d69f334524d43bba3515f98e5ab37: Log is configured to *not* fsync() on all Append() calls
I20260926 03:47:06.402338 14358 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 129d69f334524d43bba3515f98e5ab37: No bootstrap required, opened a new log
W20260926 03:47:06.407611 14386 tablet.cc:2369] T 00000000000000000000000000000000 P 64062209971045e0b7737c1e8d356abb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 03:47:06.414002 14358 raft_consensus.cc:348] T 00000000000000000000000000000000 P 129d69f334524d43bba3515f98e5ab37 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a42cb64b376e49b6af88f02210fb9d2d" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 41069 } } peers { permanent_uuid: "129d69f334524d43bba3515f98e5ab37" member_type: VOTER last_known_addr { host: "127.0.194.253" port: 45537 } } peers { permanent_uuid: "64062209971045e0b7737c1e8d356abb" member_type: VOTER last_known_addr { host: "127.0.194.252" port: 34233 } }
I20260926 03:47:06.414460 14358 raft_consensus.cc:374] T 00000000000000000000000000000000 P 129d69f334524d43bba3515f98e5ab37 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:47:06.414577 14358 raft_consensus.cc:729] T 00000000000000000000000000000000 P 129d69f334524d43bba3515f98e5ab37 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 129d69f334524d43bba3515f98e5ab37, State: Initialized, Role: FOLLOWER
I20260926 03:47:06.415308 14358 consensus_queue.cc:260] T 00000000000000000000000000000000 P 129d69f334524d43bba3515f98e5ab37 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a42cb64b376e49b6af88f02210fb9d2d" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 41069 } } peers { permanent_uuid: "129d69f334524d43bba3515f98e5ab37" member_type: VOTER last_known_addr { host: "127.0.194.253" port: 45537 } } peers { permanent_uuid: "64062209971045e0b7737c1e8d356abb" member_type: VOTER last_known_addr { host: "127.0.194.252" port: 34233 } }
I20260926 03:47:06.416352 14475 sys_catalog.cc:455] T 00000000000000000000000000000000 P 129d69f334524d43bba3515f98e5ab37 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a42cb64b376e49b6af88f02210fb9d2d" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 41069 } } peers { permanent_uuid: "129d69f334524d43bba3515f98e5ab37" member_type: VOTER last_known_addr { host: "127.0.194.253" port: 45537 } } peers { permanent_uuid: "64062209971045e0b7737c1e8d356abb" member_type: VOTER last_known_addr { host: "127.0.194.252" port: 34233 } } }
I20260926 03:47:06.416843 14475 sys_catalog.cc:458] T 00000000000000000000000000000000 P 129d69f334524d43bba3515f98e5ab37 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 03:47:06.417811 14358 sys_catalog.cc:564] T 00000000000000000000000000000000 P 129d69f334524d43bba3515f98e5ab37 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:47:06.437697 14291 master_runner.cc:263] No leader master found from master 129d69f334524d43bba3515f98e5ab37
I20260926 03:47:06.437862 14291 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260926 03:47:06.441598 14486 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 129d69f334524d43bba3515f98e5ab37: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 03:47:06.441776 14486 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 03:47:06.542325 14447 raft_consensus.cc:482] T 00000000000000000000000000000000 P 64062209971045e0b7737c1e8d356abb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 03:47:06.542680 14447 raft_consensus.cc:504] T 00000000000000000000000000000000 P 64062209971045e0b7737c1e8d356abb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a42cb64b376e49b6af88f02210fb9d2d" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 41069 } } peers { permanent_uuid: "129d69f334524d43bba3515f98e5ab37" member_type: VOTER last_known_addr { host: "127.0.194.253" port: 45537 } } peers { permanent_uuid: "64062209971045e0b7737c1e8d356abb" member_type: VOTER last_known_addr { host: "127.0.194.252" port: 34233 } }
I20260926 03:47:06.544332 14447 leader_election.cc:288] T 00000000000000000000000000000000 P 64062209971045e0b7737c1e8d356abb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a42cb64b376e49b6af88f02210fb9d2d (127.0.194.254:41069), 129d69f334524d43bba3515f98e5ab37 (127.0.194.253:45537)
I20260926 03:47:06.545588 14259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "64062209971045e0b7737c1e8d356abb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a42cb64b376e49b6af88f02210fb9d2d" is_pre_election: true
I20260926 03:47:06.546208 14259 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a42cb64b376e49b6af88f02210fb9d2d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64062209971045e0b7737c1e8d356abb in term 0.
I20260926 03:47:06.547048 14384 leader_election.cc:302] T 00000000000000000000000000000000 P 64062209971045e0b7737c1e8d356abb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64062209971045e0b7737c1e8d356abb, a42cb64b376e49b6af88f02210fb9d2d; no voters:
I20260926 03:47:06.548182 14333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "64062209971045e0b7737c1e8d356abb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "129d69f334524d43bba3515f98e5ab37" is_pre_election: true
I20260926 03:47:06.548813 14333 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 129d69f334524d43bba3515f98e5ab37 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64062209971045e0b7737c1e8d356abb in term 0.
I20260926 03:47:06.549883 14447 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 64062209971045e0b7737c1e8d356abb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 03:47:06.550038 14447 raft_consensus.cc:482] T 00000000000000000000000000000000 P 64062209971045e0b7737c1e8d356abb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 03:47:06.550160 14447 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 64062209971045e0b7737c1e8d356abb [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:47:06.553565 14447 raft_consensus.cc:504] T 00000000000000000000000000000000 P 64062209971045e0b7737c1e8d356abb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a42cb64b376e49b6af88f02210fb9d2d" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 41069 } } peers { permanent_uuid: "129d69f334524d43bba3515f98e5ab37" member_type: VOTER last_known_addr { host: "127.0.194.253" port: 45537 } } peers { permanent_uuid: "64062209971045e0b7737c1e8d356abb" member_type: VOTER last_known_addr { host: "127.0.194.252" port: 34233 } }
I20260926 03:47:06.554945 14259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "64062209971045e0b7737c1e8d356abb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a42cb64b376e49b6af88f02210fb9d2d"
I20260926 03:47:06.555191 14259 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a42cb64b376e49b6af88f02210fb9d2d [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:47:06.558641 14259 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a42cb64b376e49b6af88f02210fb9d2d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64062209971045e0b7737c1e8d356abb in term 1.
I20260926 03:47:06.559237 14384 leader_election.cc:302] T 00000000000000000000000000000000 P 64062209971045e0b7737c1e8d356abb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64062209971045e0b7737c1e8d356abb, a42cb64b376e49b6af88f02210fb9d2d; no voters:
I20260926 03:47:06.561540 14447 leader_election.cc:288] T 00000000000000000000000000000000 P 64062209971045e0b7737c1e8d356abb [CANDIDATE]: Term 1 election: Requested vote from peers a42cb64b376e49b6af88f02210fb9d2d (127.0.194.254:41069), 129d69f334524d43bba3515f98e5ab37 (127.0.194.253:45537)
I20260926 03:47:06.566280 14447 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 64062209971045e0b7737c1e8d356abb [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:47:06.566777 14447 raft_consensus.cc:686] T 00000000000000000000000000000000 P 64062209971045e0b7737c1e8d356abb [term 1 LEADER]: Becoming Leader. State: Replica: 64062209971045e0b7737c1e8d356abb, State: Running, Role: LEADER
I20260926 03:47:06.567569 14447 consensus_queue.cc:237] T 00000000000000000000000000000000 P 64062209971045e0b7737c1e8d356abb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a42cb64b376e49b6af88f02210fb9d2d" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 41069 } } peers { permanent_uuid: "129d69f334524d43bba3515f98e5ab37" member_type: VOTER last_known_addr { host: "127.0.194.253" port: 45537 } } peers { permanent_uuid: "64062209971045e0b7737c1e8d356abb" member_type: VOTER last_known_addr { host: "127.0.194.252" port: 34233 } }
I20260926 03:47:06.577833 14333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "64062209971045e0b7737c1e8d356abb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "129d69f334524d43bba3515f98e5ab37"
I20260926 03:47:06.578140 14333 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 129d69f334524d43bba3515f98e5ab37 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:47:06.581645 14333 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 129d69f334524d43bba3515f98e5ab37 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64062209971045e0b7737c1e8d356abb in term 1.
I20260926 03:47:06.583774 14488 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64062209971045e0b7737c1e8d356abb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 64062209971045e0b7737c1e8d356abb. Latest consensus state: current_term: 1 leader_uuid: "64062209971045e0b7737c1e8d356abb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a42cb64b376e49b6af88f02210fb9d2d" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 41069 } } peers { permanent_uuid: "129d69f334524d43bba3515f98e5ab37" member_type: VOTER last_known_addr { host: "127.0.194.253" port: 45537 } } peers { permanent_uuid: "64062209971045e0b7737c1e8d356abb" member_type: VOTER last_known_addr { host: "127.0.194.252" port: 34233 } } }
I20260926 03:47:06.584353 14488 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64062209971045e0b7737c1e8d356abb [sys.catalog]: This master's current role is: LEADER
I20260926 03:47:06.596293 14490 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:47:06.598876 14490 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:47:06.619961 14333 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 129d69f334524d43bba3515f98e5ab37 [term 1 FOLLOWER]: Refusing update from remote peer 64062209971045e0b7737c1e8d356abb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:47:06.621232 14488 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 64062209971045e0b7737c1e8d356abb [LEADER]: Connected to new peer: Peer: permanent_uuid: "129d69f334524d43bba3515f98e5ab37" member_type: VOTER last_known_addr { host: "127.0.194.253" port: 45537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 03:47:06.622298 14312 tablet.cc:2369] T 00000000000000000000000000000000 P 129d69f334524d43bba3515f98e5ab37: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 03:47:06.628998 14259 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a42cb64b376e49b6af88f02210fb9d2d [term 1 FOLLOWER]: Refusing update from remote peer 64062209971045e0b7737c1e8d356abb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:47:06.629909 14488 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 64062209971045e0b7737c1e8d356abb [LEADER]: Connected to new peer: Peer: permanent_uuid: "a42cb64b376e49b6af88f02210fb9d2d" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 41069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:47:06.669374 14461 sys_catalog.cc:455] T 00000000000000000000000000000000 P a42cb64b376e49b6af88f02210fb9d2d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 64062209971045e0b7737c1e8d356abb. Latest consensus state: current_term: 1 leader_uuid: "64062209971045e0b7737c1e8d356abb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a42cb64b376e49b6af88f02210fb9d2d" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 41069 } } peers { permanent_uuid: "129d69f334524d43bba3515f98e5ab37" member_type: VOTER last_known_addr { host: "127.0.194.253" port: 45537 } } peers { permanent_uuid: "64062209971045e0b7737c1e8d356abb" member_type: VOTER last_known_addr { host: "127.0.194.252" port: 34233 } } }
I20260926 03:47:06.670395 14461 sys_catalog.cc:458] T 00000000000000000000000000000000 P a42cb64b376e49b6af88f02210fb9d2d [sys.catalog]: This master's current role is: FOLLOWER
I20260926 03:47:06.670517 14491 mvcc.cc:205] Tried to move back new op lower bound from 7333455571423129600 to 7333455571231145984. Current Snapshot: MvccSnapshot[applied={T|T < 7333455571423129600}]
I20260926 03:47:06.672587 14493 mvcc.cc:205] Tried to move back new op lower bound from 7333455571423129600 to 7333455571231145984. Current Snapshot: MvccSnapshot[applied={T|T < 7333455571423129600}]
I20260926 03:47:06.673336 14490 catalog_manager.cc:1347] Generated new cluster ID: a0423ac5873345db97e0bea19a1bfcd7
I20260926 03:47:06.673448 14490 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:47:06.673537 14488 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64062209971045e0b7737c1e8d356abb [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "64062209971045e0b7737c1e8d356abb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a42cb64b376e49b6af88f02210fb9d2d" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 41069 } } peers { permanent_uuid: "129d69f334524d43bba3515f98e5ab37" member_type: VOTER last_known_addr { host: "127.0.194.253" port: 45537 } } peers { permanent_uuid: "64062209971045e0b7737c1e8d356abb" member_type: VOTER last_known_addr { host: "127.0.194.252" port: 34233 } } }
I20260926 03:47:06.673818 14488 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64062209971045e0b7737c1e8d356abb [sys.catalog]: This master's current role is: LEADER
I20260926 03:47:06.687366 14461 sys_catalog.cc:455] T 00000000000000000000000000000000 P a42cb64b376e49b6af88f02210fb9d2d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "64062209971045e0b7737c1e8d356abb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a42cb64b376e49b6af88f02210fb9d2d" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 41069 } } peers { permanent_uuid: "129d69f334524d43bba3515f98e5ab37" member_type: VOTER last_known_addr { host: "127.0.194.253" port: 45537 } } peers { permanent_uuid: "64062209971045e0b7737c1e8d356abb" member_type: VOTER last_known_addr { host: "127.0.194.252" port: 34233 } } }
I20260926 03:47:06.687702 14461 sys_catalog.cc:458] T 00000000000000000000000000000000 P a42cb64b376e49b6af88f02210fb9d2d [sys.catalog]: This master's current role is: FOLLOWER
I20260926 03:47:06.692169 14475 sys_catalog.cc:455] T 00000000000000000000000000000000 P 129d69f334524d43bba3515f98e5ab37 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 64062209971045e0b7737c1e8d356abb. Latest consensus state: current_term: 1 leader_uuid: "64062209971045e0b7737c1e8d356abb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a42cb64b376e49b6af88f02210fb9d2d" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 41069 } } peers { permanent_uuid: "129d69f334524d43bba3515f98e5ab37" member_type: VOTER last_known_addr { host: "127.0.194.253" port: 45537 } } peers { permanent_uuid: "64062209971045e0b7737c1e8d356abb" member_type: VOTER last_known_addr { host: "127.0.194.252" port: 34233 } } }
I20260926 03:47:06.692539 14475 sys_catalog.cc:458] T 00000000000000000000000000000000 P 129d69f334524d43bba3515f98e5ab37 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 03:47:06.696030 14447 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64062209971045e0b7737c1e8d356abb [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "64062209971045e0b7737c1e8d356abb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a42cb64b376e49b6af88f02210fb9d2d" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 41069 } } peers { permanent_uuid: "129d69f334524d43bba3515f98e5ab37" member_type: VOTER last_known_addr { host: "127.0.194.253" port: 45537 } } peers { permanent_uuid: "64062209971045e0b7737c1e8d356abb" member_type: VOTER last_known_addr { host: "127.0.194.252" port: 34233 } } }
I20260926 03:47:06.696398 14447 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64062209971045e0b7737c1e8d356abb [sys.catalog]: This master's current role is: LEADER
I20260926 03:47:06.698521 14492 mvcc.cc:205] Tried to move back new op lower bound from 7333455571423129600 to 7333455571231145984. Current Snapshot: MvccSnapshot[applied={T|T < 7333455571423129600}]
I20260926 03:47:06.711884 14475 sys_catalog.cc:455] T 00000000000000000000000000000000 P 129d69f334524d43bba3515f98e5ab37 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "64062209971045e0b7737c1e8d356abb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a42cb64b376e49b6af88f02210fb9d2d" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 41069 } } peers { permanent_uuid: "129d69f334524d43bba3515f98e5ab37" member_type: VOTER last_known_addr { host: "127.0.194.253" port: 45537 } } peers { permanent_uuid: "64062209971045e0b7737c1e8d356abb" member_type: VOTER last_known_addr { host: "127.0.194.252" port: 34233 } } }
I20260926 03:47:06.712177 14475 sys_catalog.cc:458] T 00000000000000000000000000000000 P 129d69f334524d43bba3515f98e5ab37 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 03:47:06.715333 14490 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:47:06.718788 14490 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:47:06.737608 14490 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 64062209971045e0b7737c1e8d356abb: Generated new TSK 0
I20260926 03:47:06.738970 14490 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 03:47:07.341524 14472 catalog_manager.cc:1259] Loaded cluster ID: a0423ac5873345db97e0bea19a1bfcd7
I20260926 03:47:07.341629 14472 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a42cb64b376e49b6af88f02210fb9d2d: loading cluster ID for follower catalog manager: success
I20260926 03:47:07.344091 14472 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a42cb64b376e49b6af88f02210fb9d2d: acquiring CA information for follower catalog manager: success
I20260926 03:47:07.346073 14472 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a42cb64b376e49b6af88f02210fb9d2d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 03:47:07.443835 14486 catalog_manager.cc:1259] Loaded cluster ID: a0423ac5873345db97e0bea19a1bfcd7
I20260926 03:47:07.443979 14486 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 129d69f334524d43bba3515f98e5ab37: loading cluster ID for follower catalog manager: success
I20260926 03:47:07.446483 14486 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 129d69f334524d43bba3515f98e5ab37: acquiring CA information for follower catalog manager: success
I20260926 03:47:07.448415 14486 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 129d69f334524d43bba3515f98e5ab37: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 03:47:07.864950 779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu pid 14217
I20260926 03:47:07.866253 14279 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:47:07.976873 779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu with pid 14217
I20260926 03:47:07.997893 779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu pid 14291
I20260926 03:47:07.999181 14353 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:47:08.104954 779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu with pid 14291
I20260926 03:47:08.125936 779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu pid 14365
I20260926 03:47:08.127167 14427 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:47:08.236038 14505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:08.235322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:41069 (local address 127.0.0.1:54426)
0926 03:47:08.235761 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0926 03:47:08.235884 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.254:41069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":216,"threads_started":1}
I20260926 03:47:08.236027 779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu with pid 14365
2026-09-26T03:47:08Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4905 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790394221564982-779-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:47:08 dist-test-slave-n2th krb5kdc[14508](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:47:08 dist-test-slave-n2th krb5kdc[14508](info): set up 2 sockets
Sep 26 03:47:08 dist-test-slave-n2th krb5kdc[14508](info): commencing operation
krb5kdc: starting...
W20260926 03:47:09.339354 779 mini_kdc.cc:121] Time spent starting KDC: real 1.060s user 0.008s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790394221564982-779-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790394221564982-779-0/krb5kdc/test-user.keytab.
Sep 26 03:47:09 dist-test-slave-n2th krb5kdc[14508](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394429, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:47:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:47:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.194.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.194.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.194.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790394221564982-779-0/krb5kdc/kudu_127.0.194.254.keytab.
Entry for principal kudu/127.0.194.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790394221564982-779-0/krb5kdc/kudu_127.0.194.254.keytab.
I20260926 03:47:09.581880 779 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
/tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790394221564982-779-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790394221564982-779-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790394221564982-779-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790394221564982-779-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.194.254:34377
--webserver_interface=127.0.194.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790394221564982-779-0/krb5kdc/kudu_127.0.194.254.keytab
--principal=kudu/127.0.194.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.194.212:39275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.194.254:34377
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790394221564982-779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790394221564982-779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790394221564982-779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:47:09.925863 14521 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:47:09.926179 14521 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:47:09.926365 14521 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:47:09.933884 14521 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:47:09.934010 14521 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:47:09.934103 14521 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:47:09.934186 14521 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:47:09.934248 14521 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:47:09.944423 14521 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.194.212:39275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790394221564982-779-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790394221564982-779-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.194.254:34377
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790394221564982-779-0/krb5kdc/kudu_127.0.194.254.keytab
--principal=kudu/127.0.194.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.194.254:34377
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790394221564982-779-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.194.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790394221564982-779-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:47:09.945873 14521 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:47:09.947312 14521 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:47:09.959422 14526 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:47:09.959362 14527 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:47:09.959401 14529 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:47:09.960630 14521 server_base.cc:1034] running on GCE node
I20260926 03:47:09.961585 14521 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:47:09.963348 14521 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:47:09.964584 14521 hybrid_clock.cc:648] HybridClock initialized: now 1790394429964536 us; error 37 us; skew 500 ppm
I20260926 03:47:09.964950 14521 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:47:09 dist-test-slave-n2th krb5kdc[14508](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394429, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:47:09.968082 14521 init.cc:377] Logged in from keytab as kudu/127.0.194.254@KRBTEST.COM (short username kudu)
I20260926 03:47:09.970031 14521 webserver.cc:460] Webserver started at http://127.0.194.254:43399/ using document root <none> and password file <none>
I20260926 03:47:09.970844 14521 fs_manager.cc:359] Metadata directory not provided
I20260926 03:47:09.970983 14521 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:47:09.971309 14521 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:47:09.974304 14521 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790394221564982-779-0/minicluster-data/master-0/data/instance:
uuid: "231d21aa80c240229f9c2c172e82e8e0"
format_stamp: "Formatted at 2026-09-26 03:47:09 on dist-test-slave-n2th"
server_key: "b0d5ec1f10b5dc7e21204779f456b894"
server_key_iv: "13088bdc59c45758825d66556b6912f0"
server_key_version: "encryptionkey@0"
I20260926 03:47:09.975124 14521 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790394221564982-779-0/minicluster-data/master-0/wal/instance:
uuid: "231d21aa80c240229f9c2c172e82e8e0"
format_stamp: "Formatted at 2026-09-26 03:47:09 on dist-test-slave-n2th"
server_key: "b0d5ec1f10b5dc7e21204779f456b894"
server_key_iv: "13088bdc59c45758825d66556b6912f0"
server_key_version: "encryptionkey@0"
I20260926 03:47:09.980613 14521 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 03:47:09.984119 14536 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:47:09.985644 14521 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 03:47:09.985850 14521 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790394221564982-779-0/minicluster-data/master-0/data,/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790394221564982-779-0/minicluster-data/master-0/wal
uuid: "231d21aa80c240229f9c2c172e82e8e0"
format_stamp: "Formatted at 2026-09-26 03:47:09 on dist-test-slave-n2th"
server_key: "b0d5ec1f10b5dc7e21204779f456b894"
server_key_iv: "13088bdc59c45758825d66556b6912f0"
server_key_version: "encryptionkey@0"
I20260926 03:47:09.986069 14521 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790394221564982-779-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790394221564982-779-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790394221564982-779-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:47:10.011226 14521 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:47:10.015488 14521 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:47:10.015796 14521 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:47:10.034927 14521 rpc_server.cc:307] RPC server started. Bound to: 127.0.194.254:34377
I20260926 03:47:10.034931 14587 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.194.254:34377 every 8 connection(s)
I20260926 03:47:10.036379 14521 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790394221564982-779-0/minicluster-data/master-0/data/info.pb
I20260926 03:47:10.040150 14588 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:47:10.046478 779 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu as pid 14521
I20260926 03:47:10.046770 779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790394221564982-779-0/minicluster-data/master-0/wal/instance
I20260926 03:47:10.047665 779 external_mini_cluster.cc:1428] Setting key 9affc6353a9ff6540b0a6d53de7c92be
I20260926 03:47:10.051396 14588 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 231d21aa80c240229f9c2c172e82e8e0: Bootstrap starting.
I20260926 03:47:10.055275 14588 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 231d21aa80c240229f9c2c172e82e8e0: Neither blocks nor log segments found. Creating new log.
I20260926 03:47:10.056408 14588 log.cc:824] T 00000000000000000000000000000000 P 231d21aa80c240229f9c2c172e82e8e0: Log is configured to *not* fsync() on all Append() calls
I20260926 03:47:10.059755 14588 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 231d21aa80c240229f9c2c172e82e8e0: No bootstrap required, opened a new log
Sep 26 03:47:10 dist-test-slave-n2th krb5kdc[14508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394429, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.194.254@KRBTEST.COM
I20260926 03:47:10.068302 14588 raft_consensus.cc:348] T 00000000000000000000000000000000 P 231d21aa80c240229f9c2c172e82e8e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "231d21aa80c240229f9c2c172e82e8e0" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 34377 } }
I20260926 03:47:10.068811 14588 raft_consensus.cc:374] T 00000000000000000000000000000000 P 231d21aa80c240229f9c2c172e82e8e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:47:10.068935 14588 raft_consensus.cc:729] T 00000000000000000000000000000000 P 231d21aa80c240229f9c2c172e82e8e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 231d21aa80c240229f9c2c172e82e8e0, State: Initialized, Role: FOLLOWER
I20260926 03:47:10.069574 14588 consensus_queue.cc:260] T 00000000000000000000000000000000 P 231d21aa80c240229f9c2c172e82e8e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "231d21aa80c240229f9c2c172e82e8e0" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 34377 } }
I20260926 03:47:10.069832 14588 raft_consensus.cc:388] T 00000000000000000000000000000000 P 231d21aa80c240229f9c2c172e82e8e0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:47:10.069998 14588 raft_consensus.cc:482] T 00000000000000000000000000000000 P 231d21aa80c240229f9c2c172e82e8e0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:47:10.070199 14588 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 231d21aa80c240229f9c2c172e82e8e0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:47:10.072681 14588 raft_consensus.cc:504] T 00000000000000000000000000000000 P 231d21aa80c240229f9c2c172e82e8e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "231d21aa80c240229f9c2c172e82e8e0" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 34377 } }
I20260926 03:47:10.073304 14588 leader_election.cc:302] T 00000000000000000000000000000000 P 231d21aa80c240229f9c2c172e82e8e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 231d21aa80c240229f9c2c172e82e8e0; no voters:
I20260926 03:47:10.073976 14588 leader_election.cc:288] T 00000000000000000000000000000000 P 231d21aa80c240229f9c2c172e82e8e0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:47:10.074115 14593 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 231d21aa80c240229f9c2c172e82e8e0 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:47:10.075450 14593 raft_consensus.cc:686] T 00000000000000000000000000000000 P 231d21aa80c240229f9c2c172e82e8e0 [term 1 LEADER]: Becoming Leader. State: Replica: 231d21aa80c240229f9c2c172e82e8e0, State: Running, Role: LEADER
I20260926 03:47:10.076136 14593 consensus_queue.cc:237] T 00000000000000000000000000000000 P 231d21aa80c240229f9c2c172e82e8e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "231d21aa80c240229f9c2c172e82e8e0" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 34377 } }
I20260926 03:47:10.076546 14588 sys_catalog.cc:564] T 00000000000000000000000000000000 P 231d21aa80c240229f9c2c172e82e8e0 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:47:10.077003 14591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:10.049574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55514 (local address 127.0.194.254:34377)
0926 03:47:10.050384 (+ 810us) server_negotiation.cc:206] Beginning negotiation
0926 03:47:10.050396 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 03:47:10.050437 (+ 41us) server_negotiation.cc:407] Connection header received
0926 03:47:10.052330 (+ 1893us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:47:10.052354 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:47:10.052784 (+ 430us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:47:10.053161 (+ 377us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:47:10.055194 (+ 2033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:47:10.056634 (+ 1440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:47:10.058284 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:47:10.059072 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:47:10.059213 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:47:10.064298 (+ 5085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:47:10.064306 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:47:10.064323 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:47:10.064375 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:47:10.071711 (+ 7336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:47:10.072595 (+ 884us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:47:10.072605 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:47:10.072615 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:47:10.072843 (+ 228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:47:10.073297 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:47:10.073303 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:47:10.073306 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:47:10.073957 (+ 651us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:47:10.074308 (+ 351us) server_negotiation.cc:1035] Waiting for connection context
0926 03:47:10.074983 (+ 675us) server_negotiation.cc:299] Negotiation successful
0926 03:47:10.075552 (+ 569us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":594,"thread_start_us":372,"threads_started":1}
I20260926 03:47:10.079452 14595 sys_catalog.cc:455] T 00000000000000000000000000000000 P 231d21aa80c240229f9c2c172e82e8e0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 231d21aa80c240229f9c2c172e82e8e0. Latest consensus state: current_term: 1 leader_uuid: "231d21aa80c240229f9c2c172e82e8e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "231d21aa80c240229f9c2c172e82e8e0" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 34377 } } }
I20260926 03:47:10.079516 14594 sys_catalog.cc:455] T 00000000000000000000000000000000 P 231d21aa80c240229f9c2c172e82e8e0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "231d21aa80c240229f9c2c172e82e8e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "231d21aa80c240229f9c2c172e82e8e0" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 34377 } } }
I20260926 03:47:10.079872 14594 sys_catalog.cc:458] T 00000000000000000000000000000000 P 231d21aa80c240229f9c2c172e82e8e0 [sys.catalog]: This master's current role is: LEADER
I20260926 03:47:10.079869 14595 sys_catalog.cc:458] T 00000000000000000000000000000000 P 231d21aa80c240229f9c2c172e82e8e0 [sys.catalog]: This master's current role is: LEADER
I20260926 03:47:10.082263 779 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 03:47:10.089413 14598 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:47:10.094864 14598 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:47:10.103873 14598 catalog_manager.cc:1347] Generated new cluster ID: d52b2024e0414580afbb3320347e6c02
I20260926 03:47:10.104019 14598 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:47:10.116093 14598 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:47:10.117511 14598 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:47:10.125856 14598 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 231d21aa80c240229f9c2c172e82e8e0: Generated new TSK 0
I20260926 03:47:10.126844 14598 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 03:47:10.145090 779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu pid 14521
I20260926 03:47:10.146368 14583 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:47:10.239097 779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu with pid 14521
2026-09-26T03:47:10Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1997 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:47:10 dist-test-slave-n2th krb5kdc[14616](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:47:10 dist-test-slave-n2th krb5kdc[14616](info): set up 2 sockets
Sep 26 03:47:10 dist-test-slave-n2th krb5kdc[14616](info): commencing operation
krb5kdc: starting...
W20260926 03:47:11.343228 779 mini_kdc.cc:121] Time spent starting KDC: real 1.067s user 0.005s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/krb5kdc/test-user.keytab.
Sep 26 03:47:11 dist-test-slave-n2th krb5kdc[14616](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394431, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:47:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:47:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.194.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.194.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.194.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/krb5kdc/kudu_127.0.194.254.keytab.
Entry for principal kudu/127.0.194.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/krb5kdc/kudu_127.0.194.254.keytab.
I20260926 03:47:11.598217 779 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
/tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.194.254:34561
--webserver_interface=127.0.194.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/krb5kdc/kudu_127.0.194.254.keytab
--principal=kudu/127.0.194.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.194.212:45229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.194.254:34561
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:47:11.938230 14629 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:47:11.938598 14629 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:47:11.938797 14629 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:47:11.946169 14629 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:47:11.946337 14629 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:47:11.946457 14629 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:47:11.946557 14629 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:47:11.946663 14629 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:47:11.956771 14629 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.194.212:45229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.194.254:34561
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/krb5kdc/kudu_127.0.194.254.keytab
--principal=kudu/127.0.194.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.194.254:34561
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.194.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:47:11.958170 14629 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:47:11.959610 14629 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:47:11.971521 14634 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:47:11.971490 14637 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:47:11.971573 14635 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:47:11.972682 14629 server_base.cc:1034] running on GCE node
I20260926 03:47:11.973783 14629 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:47:11.975346 14629 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:47:11.976550 14629 hybrid_clock.cc:648] HybridClock initialized: now 1790394431976512 us; error 32 us; skew 500 ppm
I20260926 03:47:11.976899 14629 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:47:11 dist-test-slave-n2th krb5kdc[14616](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394431, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:47:11.980106 14629 init.cc:377] Logged in from keytab as kudu/127.0.194.254@KRBTEST.COM (short username kudu)
I20260926 03:47:11.981699 14629 webserver.cc:460] Webserver started at http://127.0.194.254:37679/ using document root <none> and password file <none>
I20260926 03:47:11.982420 14629 fs_manager.cc:359] Metadata directory not provided
I20260926 03:47:11.982571 14629 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:47:11.982944 14629 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:47:11.986018 14629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/minicluster-data/master-0/data/instance:
uuid: "fd19af12fdda4aa9be763b660e4611c8"
format_stamp: "Formatted at 2026-09-26 03:47:11 on dist-test-slave-n2th"
server_key: "a8703d5d0686fe8f2af4d77f386693cf"
server_key_iv: "571485da51254437ea71931d7e4ab0d3"
server_key_version: "encryptionkey@0"
I20260926 03:47:11.986954 14629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/minicluster-data/master-0/wal/instance:
uuid: "fd19af12fdda4aa9be763b660e4611c8"
format_stamp: "Formatted at 2026-09-26 03:47:11 on dist-test-slave-n2th"
server_key: "a8703d5d0686fe8f2af4d77f386693cf"
server_key_iv: "571485da51254437ea71931d7e4ab0d3"
server_key_version: "encryptionkey@0"
I20260926 03:47:11.992403 14629 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 03:47:11.995982 14644 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:47:11.997361 14629 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 03:47:11.997560 14629 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/minicluster-data/master-0/data,/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/minicluster-data/master-0/wal
uuid: "fd19af12fdda4aa9be763b660e4611c8"
format_stamp: "Formatted at 2026-09-26 03:47:11 on dist-test-slave-n2th"
server_key: "a8703d5d0686fe8f2af4d77f386693cf"
server_key_iv: "571485da51254437ea71931d7e4ab0d3"
server_key_version: "encryptionkey@0"
I20260926 03:47:11.997778 14629 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:47:12.019452 14629 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:47:12.023717 14629 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:47:12.024029 14629 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:47:12.042713 14629 rpc_server.cc:307] RPC server started. Bound to: 127.0.194.254:34561
I20260926 03:47:12.042778 14695 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.194.254:34561 every 8 connection(s)
I20260926 03:47:12.044200 14629 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/minicluster-data/master-0/data/info.pb
I20260926 03:47:12.048101 14696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:47:12.052886 779 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu as pid 14629
I20260926 03:47:12.053133 779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/minicluster-data/master-0/wal/instance
I20260926 03:47:12.053988 779 external_mini_cluster.cc:1428] Setting key 825a17772cacd4a500defd55124cb9e5
I20260926 03:47:12.060499 14696 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd19af12fdda4aa9be763b660e4611c8: Bootstrap starting.
I20260926 03:47:12.064503 14696 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fd19af12fdda4aa9be763b660e4611c8: Neither blocks nor log segments found. Creating new log.
I20260926 03:47:12.065729 14696 log.cc:824] T 00000000000000000000000000000000 P fd19af12fdda4aa9be763b660e4611c8: Log is configured to *not* fsync() on all Append() calls
I20260926 03:47:12.069414 14696 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd19af12fdda4aa9be763b660e4611c8: No bootstrap required, opened a new log
Sep 26 03:47:12 dist-test-slave-n2th krb5kdc[14616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394431, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.194.254@KRBTEST.COM
I20260926 03:47:12.077510 14696 raft_consensus.cc:348] T 00000000000000000000000000000000 P fd19af12fdda4aa9be763b660e4611c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd19af12fdda4aa9be763b660e4611c8" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 34561 } }
I20260926 03:47:12.077956 14696 raft_consensus.cc:374] T 00000000000000000000000000000000 P fd19af12fdda4aa9be763b660e4611c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:47:12.078106 14696 raft_consensus.cc:729] T 00000000000000000000000000000000 P fd19af12fdda4aa9be763b660e4611c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd19af12fdda4aa9be763b660e4611c8, State: Initialized, Role: FOLLOWER
I20260926 03:47:12.078717 14696 consensus_queue.cc:260] T 00000000000000000000000000000000 P fd19af12fdda4aa9be763b660e4611c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd19af12fdda4aa9be763b660e4611c8" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 34561 } }
I20260926 03:47:12.079015 14699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:12.056234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59612 (local address 127.0.194.254:34561)
0926 03:47:12.057073 (+ 839us) server_negotiation.cc:206] Beginning negotiation
0926 03:47:12.057086 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 03:47:12.057133 (+ 47us) server_negotiation.cc:407] Connection header received
0926 03:47:12.058971 (+ 1838us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:47:12.058997 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:47:12.059405 (+ 408us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:47:12.059740 (+ 335us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:47:12.061677 (+ 1937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:47:12.063125 (+ 1448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:47:12.065048 (+ 1923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:47:12.065739 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:47:12.065961 (+ 222us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:47:12.070982 (+ 5021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:47:12.070991 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:47:12.071006 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:47:12.071047 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:47:12.074264 (+ 3217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:47:12.075114 (+ 850us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:47:12.075121 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:47:12.075128 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:47:12.075306 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:47:12.075702 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:47:12.075714 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:47:12.075716 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:47:12.076378 (+ 662us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:47:12.076708 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0926 03:47:12.077130 (+ 422us) server_negotiation.cc:299] Negotiation successful
0926 03:47:12.077579 (+ 449us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":619,"thread_start_us":407,"threads_started":1}
I20260926 03:47:12.079032 14696 raft_consensus.cc:388] T 00000000000000000000000000000000 P fd19af12fdda4aa9be763b660e4611c8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:47:12.079217 14696 raft_consensus.cc:482] T 00000000000000000000000000000000 P fd19af12fdda4aa9be763b660e4611c8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:47:12.079435 14696 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fd19af12fdda4aa9be763b660e4611c8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:47:12.081930 14696 raft_consensus.cc:504] T 00000000000000000000000000000000 P fd19af12fdda4aa9be763b660e4611c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd19af12fdda4aa9be763b660e4611c8" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 34561 } }
I20260926 03:47:12.082558 14696 leader_election.cc:302] T 00000000000000000000000000000000 P fd19af12fdda4aa9be763b660e4611c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fd19af12fdda4aa9be763b660e4611c8; no voters:
I20260926 03:47:12.083230 14696 leader_election.cc:288] T 00000000000000000000000000000000 P fd19af12fdda4aa9be763b660e4611c8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:47:12.083335 14701 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fd19af12fdda4aa9be763b660e4611c8 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:47:12.084640 14701 raft_consensus.cc:686] T 00000000000000000000000000000000 P fd19af12fdda4aa9be763b660e4611c8 [term 1 LEADER]: Becoming Leader. State: Replica: fd19af12fdda4aa9be763b660e4611c8, State: Running, Role: LEADER
I20260926 03:47:12.085199 14701 consensus_queue.cc:237] T 00000000000000000000000000000000 P fd19af12fdda4aa9be763b660e4611c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd19af12fdda4aa9be763b660e4611c8" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 34561 } }
I20260926 03:47:12.085475 14696 sys_catalog.cc:564] T 00000000000000000000000000000000 P fd19af12fdda4aa9be763b660e4611c8 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:47:12.088305 14703 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd19af12fdda4aa9be763b660e4611c8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fd19af12fdda4aa9be763b660e4611c8. Latest consensus state: current_term: 1 leader_uuid: "fd19af12fdda4aa9be763b660e4611c8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd19af12fdda4aa9be763b660e4611c8" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 34561 } } }
I20260926 03:47:12.088325 14702 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd19af12fdda4aa9be763b660e4611c8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fd19af12fdda4aa9be763b660e4611c8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd19af12fdda4aa9be763b660e4611c8" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 34561 } } }
I20260926 03:47:12.088789 14702 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd19af12fdda4aa9be763b660e4611c8 [sys.catalog]: This master's current role is: LEADER
I20260926 03:47:12.088793 14703 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd19af12fdda4aa9be763b660e4611c8 [sys.catalog]: This master's current role is: LEADER
I20260926 03:47:12.090355 14708 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:47:12.095207 14708 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:47:12.105355 14708 catalog_manager.cc:1347] Generated new cluster ID: 043e28fe1f644dc78387fad6417f56fa
I20260926 03:47:12.105538 14708 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:47:12.121567 14708 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:47:12.123059 14708 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:47:12.133703 14708 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fd19af12fdda4aa9be763b660e4611c8: Generated new TSK 0
I20260926 03:47:12.134588 14708 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.194.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.194.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.194.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/krb5kdc/kudu_127.0.194.193.keytab.
Entry for principal kudu/127.0.194.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/krb5kdc/kudu_127.0.194.193.keytab.
I20260926 03:47:12.195052 779 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
/tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.194.193:0
--local_ip_for_outbound_sockets=127.0.194.193
--webserver_interface=127.0.194.193
--webserver_port=0
--tserver_master_addrs=127.0.194.254:34561
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/krb5kdc/kudu_127.0.194.193.keytab
--principal=kudu/127.0.194.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.194.212:45229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:47:12.536289 14722 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:47:12.536641 14722 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:47:12.536813 14722 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:47:12.544260 14722 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:47:12.544432 14722 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:47:12.544625 14722 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.194.193
I20260926 03:47:12.554658 14722 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.194.212:45229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/krb5kdc/kudu_127.0.194.193.keytab
--principal=kudu/127.0.194.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.194.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.194.193
--webserver_port=0
--tserver_master_addrs=127.0.194.254:34561
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.194.193
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:47:12.556114 14722 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:47:12.557485 14722 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:47:12.571162 14727 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:47:12.571194 14728 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:47:12.571396 14730 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:47:12.572203 14722 server_base.cc:1034] running on GCE node
I20260926 03:47:12.572937 14722 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:47:12.573997 14722 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:47:12.575232 14722 hybrid_clock.cc:648] HybridClock initialized: now 1790394432575176 us; error 51 us; skew 500 ppm
I20260926 03:47:12.575574 14722 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:47:12 dist-test-slave-n2th krb5kdc[14616](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394432, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:47:12.578939 14722 init.cc:377] Logged in from keytab as kudu/127.0.194.193@KRBTEST.COM (short username kudu)
I20260926 03:47:12.580741 14722 webserver.cc:460] Webserver started at http://127.0.194.193:45389/ using document root <none> and password file <none>
I20260926 03:47:12.581590 14722 fs_manager.cc:359] Metadata directory not provided
I20260926 03:47:12.581751 14722 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:47:12.582094 14722 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:47:12.585319 14722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/minicluster-data/ts-0/data/instance:
uuid: "26e7194486b140e2b7434d73bbe399e6"
format_stamp: "Formatted at 2026-09-26 03:47:12 on dist-test-slave-n2th"
server_key: "00e3ccacf58a1416f41a36bf1468fda4"
server_key_iv: "68edea37252c95df0e14a621e3c1f2fb"
server_key_version: "encryptionkey@0"
I20260926 03:47:12.586234 14722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/minicluster-data/ts-0/wal/instance:
uuid: "26e7194486b140e2b7434d73bbe399e6"
format_stamp: "Formatted at 2026-09-26 03:47:12 on dist-test-slave-n2th"
server_key: "00e3ccacf58a1416f41a36bf1468fda4"
server_key_iv: "68edea37252c95df0e14a621e3c1f2fb"
server_key_version: "encryptionkey@0"
I20260926 03:47:12.592062 14722 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260926 03:47:12.595877 14737 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:47:12.597240 14722 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 03:47:12.597437 14722 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/minicluster-data/ts-0/wal
uuid: "26e7194486b140e2b7434d73bbe399e6"
format_stamp: "Formatted at 2026-09-26 03:47:12 on dist-test-slave-n2th"
server_key: "00e3ccacf58a1416f41a36bf1468fda4"
server_key_iv: "68edea37252c95df0e14a621e3c1f2fb"
server_key_version: "encryptionkey@0"
I20260926 03:47:12.597662 14722 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:47:12.619179 14722 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:47:12.623306 14722 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:47:12.623616 14722 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:47:12.624686 14722 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:47:12.626062 14722 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:47:12.626165 14722 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:47:12.626315 14722 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:47:12.626389 14722 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:47:12.660975 14722 rpc_server.cc:307] RPC server started. Bound to: 127.0.194.193:38899
I20260926 03:47:12.660986 14849 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.194.193:38899 every 8 connection(s)
I20260926 03:47:12.662349 14722 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/minicluster-data/ts-0/data/info.pb
I20260926 03:47:12.671355 779 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu as pid 14722
I20260926 03:47:12.671602 779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/minicluster-data/ts-0/wal/instance
I20260926 03:47:12.672245 779 external_mini_cluster.cc:1428] Setting key 2ac9e686dfa03e3cde301c953e42d78e
Sep 26 03:47:12 dist-test-slave-n2th krb5kdc[14616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394432, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.193@KRBTEST.COM for kudu/127.0.194.254@KRBTEST.COM
I20260926 03:47:12.683087 14852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:12.665130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.193:40147 (local address 127.0.194.254:34561)
0926 03:47:12.665584 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0926 03:47:12.665588 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:47:12.666498 (+ 910us) server_negotiation.cc:407] Connection header received
0926 03:47:12.668476 (+ 1978us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:47:12.668483 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:47:12.668578 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:47:12.668699 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:47:12.670228 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:47:12.671028 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:47:12.672781 (+ 1753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:47:12.673333 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:47:12.673479 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:47:12.677269 (+ 3790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:47:12.677275 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:47:12.677284 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:47:12.677320 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:47:12.679911 (+ 2591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:47:12.680704 (+ 793us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:47:12.680710 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:47:12.680714 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:47:12.680845 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:47:12.681293 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:47:12.681299 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:47:12.681302 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:47:12.681575 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:47:12.681726 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0926 03:47:12.682461 (+ 735us) server_negotiation.cc:299] Negotiation successful
0926 03:47:12.682687 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":256,"threads_started":1}
I20260926 03:47:12.684119 14853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:12.665639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:34561 (local address 127.0.194.193:40147)
0926 03:47:12.666339 (+ 700us) negotiation.cc:107] Waiting for socket to connect
0926 03:47:12.666377 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 03:47:12.668196 (+ 1819us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:47:12.668954 (+ 758us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:47:12.668963 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:47:12.669479 (+ 516us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:47:12.670050 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:47:12.670067 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:47:12.671201 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:47:12.671205 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:47:12.672611 (+ 1406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:47:12.672624 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:47:12.673579 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:47:12.673587 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:47:12.673843 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:47:12.674620 (+ 777us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:47:12.674647 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:47:12.677024 (+ 2377us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:47:12.680102 (+ 3078us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:47:12.680112 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:47:12.680132 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:47:12.680578 (+ 446us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:47:12.680967 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:47:12.680971 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:47:12.680975 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:47:12.681169 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:47:12.681711 (+ 542us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:47:12.681719 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:47:12.682090 (+ 371us) client_negotiation.cc:769] Sending connection context
0926 03:47:12.682366 (+ 276us) client_negotiation.cc:240] Negotiation successful
0926 03:47:12.682957 (+ 591us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":270,"threads_started":1}
I20260926 03:47:12.685833 14850 heartbeater.cc:347] Connected to a master server at 127.0.194.254:34561
I20260926 03:47:12.686128 14850 heartbeater.cc:464] Registering TS with master...
I20260926 03:47:12.686960 14850 heartbeater.cc:511] Master 127.0.194.254:34561 requested a full tablet report, sending...
I20260926 03:47:12.689352 14661 ts_manager.cc:194] Registered new tserver with Master: 26e7194486b140e2b7434d73bbe399e6 (127.0.194.193:38899)
I20260926 03:47:12.691771 14661 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.194.193@KRBTEST.COM'} at 127.0.194.193:40147
WARNING: no policy specified for kudu/127.0.194.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.194.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.194.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/krb5kdc/kudu_127.0.194.194.keytab.
Entry for principal kudu/127.0.194.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/krb5kdc/kudu_127.0.194.194.keytab.
I20260926 03:47:12.726312 779 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
/tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.194.194:0
--local_ip_for_outbound_sockets=127.0.194.194
--webserver_interface=127.0.194.194
--webserver_port=0
--tserver_master_addrs=127.0.194.254:34561
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/krb5kdc/kudu_127.0.194.194.keytab
--principal=kudu/127.0.194.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.194.212:45229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:47:13.065433 14856 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:47:13.065850 14856 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:47:13.065990 14856 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:47:13.073621 14856 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:47:13.073753 14856 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:47:13.073869 14856 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.194.194
I20260926 03:47:13.084496 14856 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.194.212:45229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/krb5kdc/kudu_127.0.194.194.keytab
--principal=kudu/127.0.194.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.194.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.194.194
--webserver_port=0
--tserver_master_addrs=127.0.194.254:34561
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.194.194
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:47:13.085843 14856 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:47:13.087321 14856 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:47:13.101269 14861 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:47:13.101207 14864 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:47:13.101291 14862 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:47:13.102380 14856 server_base.cc:1034] running on GCE node
I20260926 03:47:13.103163 14856 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:47:13.104246 14856 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:47:13.105472 14856 hybrid_clock.cc:648] HybridClock initialized: now 1790394433105432 us; error 55 us; skew 500 ppm
I20260926 03:47:13.105818 14856 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:47:13 dist-test-slave-n2th krb5kdc[14616](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394433, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:47:13.109129 14856 init.cc:377] Logged in from keytab as kudu/127.0.194.194@KRBTEST.COM (short username kudu)
I20260926 03:47:13.110723 14856 webserver.cc:460] Webserver started at http://127.0.194.194:38561/ using document root <none> and password file <none>
I20260926 03:47:13.111528 14856 fs_manager.cc:359] Metadata directory not provided
I20260926 03:47:13.111680 14856 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:47:13.112002 14856 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:47:13.115068 14856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/minicluster-data/ts-1/data/instance:
uuid: "ca26e0bdcc7c4021a1ae2b833aa2695f"
format_stamp: "Formatted at 2026-09-26 03:47:13 on dist-test-slave-n2th"
server_key: "df7b7109cc6eb457606f5360d9d6bff3"
server_key_iv: "70ac76747ff308b396c2a02aa945d2ba"
server_key_version: "encryptionkey@0"
I20260926 03:47:13.115892 14856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/minicluster-data/ts-1/wal/instance:
uuid: "ca26e0bdcc7c4021a1ae2b833aa2695f"
format_stamp: "Formatted at 2026-09-26 03:47:13 on dist-test-slave-n2th"
server_key: "df7b7109cc6eb457606f5360d9d6bff3"
server_key_iv: "70ac76747ff308b396c2a02aa945d2ba"
server_key_version: "encryptionkey@0"
I20260926 03:47:13.121316 14856 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260926 03:47:13.125020 14871 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:47:13.126448 14856 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 03:47:13.126655 14856 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/minicluster-data/ts-1/wal
uuid: "ca26e0bdcc7c4021a1ae2b833aa2695f"
format_stamp: "Formatted at 2026-09-26 03:47:13 on dist-test-slave-n2th"
server_key: "df7b7109cc6eb457606f5360d9d6bff3"
server_key_iv: "70ac76747ff308b396c2a02aa945d2ba"
server_key_version: "encryptionkey@0"
I20260926 03:47:13.126932 14856 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:47:13.145134 14856 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:47:13.149374 14856 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:47:13.149668 14856 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:47:13.150645 14856 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:47:13.152047 14856 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:47:13.152149 14856 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:47:13.152319 14856 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:47:13.152418 14856 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:47:13.186049 14856 rpc_server.cc:307] RPC server started. Bound to: 127.0.194.194:34653
I20260926 03:47:13.186143 14983 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.194.194:34653 every 8 connection(s)
I20260926 03:47:13.187479 14856 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/minicluster-data/ts-1/data/info.pb
I20260926 03:47:13.189591 779 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu as pid 14856
I20260926 03:47:13.189795 779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/minicluster-data/ts-1/wal/instance
I20260926 03:47:13.190330 779 external_mini_cluster.cc:1428] Setting key f5515b23e6449e7d4a45794af3fc95d9
Sep 26 03:47:13 dist-test-slave-n2th krb5kdc[14616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394433, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.194@KRBTEST.COM for kudu/127.0.194.254@KRBTEST.COM
I20260926 03:47:13.212378 14986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:13.190273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.194:37691 (local address 127.0.194.254:34561)
0926 03:47:13.191118 (+ 845us) server_negotiation.cc:206] Beginning negotiation
0926 03:47:13.191124 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:47:13.192018 (+ 894us) server_negotiation.cc:407] Connection header received
0926 03:47:13.194682 (+ 2664us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:47:13.194686 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:47:13.194813 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:47:13.194915 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:47:13.197028 (+ 2113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:47:13.197741 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:47:13.200354 (+ 2613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:47:13.200833 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:47:13.200934 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:47:13.205093 (+ 4159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:47:13.205103 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:47:13.205111 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:47:13.205158 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:47:13.208888 (+ 3730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:47:13.209656 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:47:13.209664 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:47:13.209671 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:47:13.209832 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:47:13.210242 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:47:13.210250 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:47:13.210256 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:47:13.210653 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:47:13.210837 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0926 03:47:13.211553 (+ 716us) server_negotiation.cc:299] Negotiation successful
0926 03:47:13.211832 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":703,"thread_start_us":572,"threads_started":1}
I20260926 03:47:13.213027 14987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:13.190939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:34561 (local address 127.0.194.194:37691)
0926 03:47:13.191866 (+ 927us) negotiation.cc:107] Waiting for socket to connect
0926 03:47:13.191913 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 03:47:13.194379 (+ 2466us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:47:13.195188 (+ 809us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:47:13.195202 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:47:13.196000 (+ 798us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:47:13.196845 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:47:13.196872 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:47:13.197903 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:47:13.197912 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:47:13.200123 (+ 2211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:47:13.200141 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:47:13.201087 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:47:13.201097 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:47:13.201448 (+ 351us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:47:13.202230 (+ 782us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:47:13.202273 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:47:13.204828 (+ 2555us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:47:13.209050 (+ 4222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:47:13.209057 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:47:13.209071 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:47:13.209509 (+ 438us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:47:13.209940 (+ 431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:47:13.209943 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:47:13.209945 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:47:13.210123 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:47:13.210824 (+ 701us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:47:13.210830 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:47:13.211157 (+ 327us) client_negotiation.cc:769] Sending connection context
0926 03:47:13.211404 (+ 247us) client_negotiation.cc:240] Negotiation successful
0926 03:47:13.211807 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":576,"thread_start_us":361,"threads_started":1}
I20260926 03:47:13.214632 14984 heartbeater.cc:347] Connected to a master server at 127.0.194.254:34561
I20260926 03:47:13.214967 14984 heartbeater.cc:464] Registering TS with master...
I20260926 03:47:13.215703 14984 heartbeater.cc:511] Master 127.0.194.254:34561 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.194.195@KRBTEST.COM; defaulting to no policy
I20260926 03:47:13.217208 14661 ts_manager.cc:194] Registered new tserver with Master: ca26e0bdcc7c4021a1ae2b833aa2695f (127.0.194.194:34653)
I20260926 03:47:13.218052 14661 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.194.194@KRBTEST.COM'} at 127.0.194.194:37691
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.194.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.194.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/krb5kdc/kudu_127.0.194.195.keytab.
Entry for principal kudu/127.0.194.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/krb5kdc/kudu_127.0.194.195.keytab.
I20260926 03:47:13.246479 779 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
/tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.194.195:0
--local_ip_for_outbound_sockets=127.0.194.195
--webserver_interface=127.0.194.195
--webserver_port=0
--tserver_master_addrs=127.0.194.254:34561
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/krb5kdc/kudu_127.0.194.195.keytab
--principal=kudu/127.0.194.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.194.212:45229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:47:13.583283 14990 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:47:13.583636 14990 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:47:13.583813 14990 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:47:13.591411 14990 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:47:13.591571 14990 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:47:13.591742 14990 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.194.195
I20260926 03:47:13.602172 14990 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.194.212:45229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/krb5kdc/kudu_127.0.194.195.keytab
--principal=kudu/127.0.194.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.194.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.194.195
--webserver_port=0
--tserver_master_addrs=127.0.194.254:34561
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.194.195
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:47:13.603639 14990 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:47:13.605026 14990 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:47:13.619060 14995 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:47:13.618980 14998 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:47:13.619025 14996 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:47:13.619931 14990 server_base.cc:1034] running on GCE node
I20260926 03:47:13.620687 14990 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:47:13.621747 14990 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:47:13.622995 14990 hybrid_clock.cc:648] HybridClock initialized: now 1790394433622943 us; error 48 us; skew 500 ppm
I20260926 03:47:13.623360 14990 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:47:13 dist-test-slave-n2th krb5kdc[14616](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394433, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:47:13.626889 14990 init.cc:377] Logged in from keytab as kudu/127.0.194.195@KRBTEST.COM (short username kudu)
I20260926 03:47:13.628575 14990 webserver.cc:460] Webserver started at http://127.0.194.195:33033/ using document root <none> and password file <none>
I20260926 03:47:13.629338 14990 fs_manager.cc:359] Metadata directory not provided
I20260926 03:47:13.629491 14990 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:47:13.629823 14990 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:47:13.632942 14990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/minicluster-data/ts-2/data/instance:
uuid: "453de609dea34f9eb8f70055f5398b7e"
format_stamp: "Formatted at 2026-09-26 03:47:13 on dist-test-slave-n2th"
server_key: "5bcb29a21950bef073643bfa6cba164f"
server_key_iv: "da0f90276eff00af416a0a415022a12d"
server_key_version: "encryptionkey@0"
I20260926 03:47:13.633790 14990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/minicluster-data/ts-2/wal/instance:
uuid: "453de609dea34f9eb8f70055f5398b7e"
format_stamp: "Formatted at 2026-09-26 03:47:13 on dist-test-slave-n2th"
server_key: "5bcb29a21950bef073643bfa6cba164f"
server_key_iv: "da0f90276eff00af416a0a415022a12d"
server_key_version: "encryptionkey@0"
I20260926 03:47:13.639364 14990 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260926 03:47:13.643383 15005 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:47:13.644896 14990 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 03:47:13.645107 14990 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/minicluster-data/ts-2/wal
uuid: "453de609dea34f9eb8f70055f5398b7e"
format_stamp: "Formatted at 2026-09-26 03:47:13 on dist-test-slave-n2th"
server_key: "5bcb29a21950bef073643bfa6cba164f"
server_key_iv: "da0f90276eff00af416a0a415022a12d"
server_key_version: "encryptionkey@0"
I20260926 03:47:13.645347 14990 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:47:13.671271 14990 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:47:13.675410 14990 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:47:13.675704 14990 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:47:13.677032 14990 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:47:13.678527 14990 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:47:13.678639 14990 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:47:13.678833 14990 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:47:13.678916 14990 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:47:13.695291 14850 heartbeater.cc:503] Master 127.0.194.254:34561 was elected leader, sending a full tablet report...
I20260926 03:47:13.716820 14990 rpc_server.cc:307] RPC server started. Bound to: 127.0.194.195:33219
I20260926 03:47:13.716917 15117 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.194.195:33219 every 8 connection(s)
I20260926 03:47:13.718374 14990 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/minicluster-data/ts-2/data/info.pb
I20260926 03:47:13.721661 779 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu as pid 14990
I20260926 03:47:13.721860 779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/minicluster-data/ts-2/wal/instance
I20260926 03:47:13.722421 779 external_mini_cluster.cc:1428] Setting key 71e10388337a94da594e11d046903c65
Sep 26 03:47:13 dist-test-slave-n2th krb5kdc[14616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394433, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.195@KRBTEST.COM for kudu/127.0.194.254@KRBTEST.COM
I20260926 03:47:13.739929 15120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:13.721932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.195:57145 (local address 127.0.194.254:34561)
0926 03:47:13.723067 (+ 1135us) server_negotiation.cc:206] Beginning negotiation
0926 03:47:13.723072 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:47:13.723824 (+ 752us) server_negotiation.cc:407] Connection header received
0926 03:47:13.725924 (+ 2100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:47:13.725928 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:47:13.725995 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:47:13.726084 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:47:13.727753 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:47:13.728513 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:47:13.730003 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:47:13.730485 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:47:13.730596 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:47:13.734069 (+ 3473us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:47:13.734074 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:47:13.734078 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:47:13.734111 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:47:13.736800 (+ 2689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:47:13.737696 (+ 896us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:47:13.737702 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:47:13.737706 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:47:13.737862 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:47:13.738330 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:47:13.738334 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:47:13.738336 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:47:13.738617 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:47:13.738796 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0926 03:47:13.739481 (+ 685us) server_negotiation.cc:299] Negotiation successful
0926 03:47:13.739649 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":984,"thread_start_us":729,"threads_started":1}
I20260926 03:47:13.740885 15121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:13.722781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:34561 (local address 127.0.194.195:57145)
0926 03:47:13.723662 (+ 881us) negotiation.cc:107] Waiting for socket to connect
0926 03:47:13.723709 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 03:47:13.725588 (+ 1879us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:47:13.726276 (+ 688us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:47:13.726285 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:47:13.727001 (+ 716us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:47:13.727598 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:47:13.727617 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:47:13.728643 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:47:13.728647 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:47:13.729876 (+ 1229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:47:13.729892 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:47:13.730726 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:47:13.730783 (+ 57us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:47:13.730998 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:47:13.731646 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:47:13.731694 (+ 48us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:47:13.733870 (+ 2176us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:47:13.736950 (+ 3080us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:47:13.736957 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:47:13.736970 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:47:13.737562 (+ 592us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:47:13.737980 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:47:13.737986 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:47:13.737989 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:47:13.738181 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:47:13.738729 (+ 548us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:47:13.738801 (+ 72us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:47:13.739131 (+ 330us) client_negotiation.cc:769] Sending connection context
0926 03:47:13.739367 (+ 236us) client_negotiation.cc:240] Negotiation successful
0926 03:47:13.739763 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":573,"thread_start_us":330,"threads_started":1}
I20260926 03:47:13.742265 15118 heartbeater.cc:347] Connected to a master server at 127.0.194.254:34561
I20260926 03:47:13.742585 15118 heartbeater.cc:464] Registering TS with master...
I20260926 03:47:13.743395 15118 heartbeater.cc:511] Master 127.0.194.254:34561 requested a full tablet report, sending...
I20260926 03:47:13.744814 14661 ts_manager.cc:194] Registered new tserver with Master: 453de609dea34f9eb8f70055f5398b7e (127.0.194.195:33219)
I20260926 03:47:13.745622 14661 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.194.195@KRBTEST.COM'} at 127.0.194.195:57145
I20260926 03:47:13.750670 779 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 03:47:13.767355 15120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:13.755132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59628 (local address 127.0.194.254:34561)
0926 03:47:13.755304 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0926 03:47:13.755308 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:47:13.755800 (+ 492us) server_negotiation.cc:407] Connection header received
0926 03:47:13.755932 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:47:13.755935 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:47:13.755998 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:47:13.756087 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:47:13.757443 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:47:13.758178 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:47:13.759581 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:47:13.759982 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:47:13.760079 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:47:13.762485 (+ 2406us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:47:13.762490 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:47:13.762493 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:47:13.762528 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:47:13.765064 (+ 2536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:47:13.765691 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:47:13.765695 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:47:13.765697 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:47:13.765833 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:47:13.766151 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:47:13.766154 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:47:13.766156 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:47:13.766461 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:47:13.766650 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0926 03:47:13.766919 (+ 269us) server_negotiation.cc:299] Negotiation successful
0926 03:47:13.767094 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 03:47:13.771425 14661 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59628:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 03:47:13.778224 14661 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 03:47:13.815477 15135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:13.790543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.194:34653 (local address 127.0.0.1:40054)
0926 03:47:13.800008 (+ 9465us) negotiation.cc:107] Waiting for socket to connect
0926 03:47:13.800076 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0926 03:47:13.800324 (+ 248us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:47:13.807240 (+ 6916us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:47:13.807252 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:47:13.807298 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:47:13.807822 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:47:13.807839 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:47:13.810085 (+ 2246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:47:13.810091 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:47:13.812299 (+ 2208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:47:13.812317 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:47:13.814305 (+ 1988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:47:13.814311 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:47:13.814526 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:47:13.814602 (+ 76us) client_negotiation.cc:769] Sending connection context
0926 03:47:13.814779 (+ 177us) client_negotiation.cc:240] Negotiation successful
0926 03:47:13.814988 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9071,"mutex_wait_us":22,"thread_start_us":14980,"threads_started":1}
I20260926 03:47:13.815701 15133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:13.790751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40054 (local address 127.0.194.194:34653)
0926 03:47:13.806088 (+ 15337us) server_negotiation.cc:206] Beginning negotiation
0926 03:47:13.806101 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 03:47:13.806142 (+ 41us) server_negotiation.cc:407] Connection header received
0926 03:47:13.806268 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:47:13.806281 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:47:13.806655 (+ 374us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:47:13.808074 (+ 1419us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:47:13.808333 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:47:13.809906 (+ 1573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:47:13.812525 (+ 2619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:47:13.814043 (+ 1518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:47:13.814232 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:47:13.814431 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0926 03:47:13.815160 (+ 729us) server_negotiation.cc:299] Negotiation successful
0926 03:47:13.815338 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15101,"thread_start_us":16110,"threads_started":1}
I20260926 03:47:13.816504 15134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:13.791185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50274 (local address 127.0.194.193:38899)
0926 03:47:13.807512 (+ 16327us) server_negotiation.cc:206] Beginning negotiation
0926 03:47:13.807519 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:47:13.807542 (+ 23us) server_negotiation.cc:407] Connection header received
0926 03:47:13.807616 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:47:13.807621 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:47:13.807793 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:47:13.807997 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:47:13.808714 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:47:13.810645 (+ 1931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:47:13.813197 (+ 2552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:47:13.814609 (+ 1412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:47:13.814792 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:47:13.815030 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0926 03:47:13.816014 (+ 984us) server_negotiation.cc:299] Negotiation successful
0926 03:47:13.816203 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16166,"thread_start_us":16006,"threads_started":1}
I20260926 03:47:13.816954 15131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:13.790717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.193:38899 (local address 127.0.0.1:50274)
0926 03:47:13.805209 (+ 14492us) negotiation.cc:107] Waiting for socket to connect
0926 03:47:13.805239 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0926 03:47:13.805415 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:47:13.807978 (+ 2563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:47:13.807982 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:47:13.808010 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:47:13.808461 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:47:13.808496 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:47:13.810828 (+ 2332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:47:13.810837 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:47:13.813021 (+ 2184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:47:13.813036 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:47:13.814890 (+ 1854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:47:13.814897 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:47:13.815101 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:47:13.815137 (+ 36us) client_negotiation.cc:769] Sending connection context
0926 03:47:13.815232 (+ 95us) client_negotiation.cc:240] Negotiation successful
0926 03:47:13.815368 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14314,"thread_start_us":18169,"threads_started":1}
I20260926 03:47:13.818185 15132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:13.792954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.195:33219 (local address 127.0.0.1:37450)
0926 03:47:13.809190 (+ 16236us) negotiation.cc:107] Waiting for socket to connect
0926 03:47:13.809209 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 03:47:13.809415 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:47:13.810455 (+ 1040us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:47:13.810462 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:47:13.810498 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:47:13.811002 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:47:13.811022 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:47:13.813527 (+ 2505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:47:13.813532 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:47:13.815759 (+ 2227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:47:13.815776 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:47:13.817417 (+ 1641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:47:13.817423 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:47:13.817584 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:47:13.817617 (+ 33us) client_negotiation.cc:769] Sending connection context
0926 03:47:13.817689 (+ 72us) client_negotiation.cc:240] Negotiation successful
0926 03:47:13.817825 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16120,"thread_start_us":6476,"threads_started":1}
I20260926 03:47:13.819208 15136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:13.793583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37450 (local address 127.0.194.195:33219)
0926 03:47:13.808740 (+ 15157us) server_negotiation.cc:206] Beginning negotiation
0926 03:47:13.808748 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:47:13.809309 (+ 561us) server_negotiation.cc:407] Connection header received
0926 03:47:13.809592 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:47:13.809601 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:47:13.810206 (+ 605us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:47:13.810378 (+ 172us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:47:13.811222 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:47:13.813374 (+ 2152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:47:13.815952 (+ 2578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:47:13.817087 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:47:13.817210 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:47:13.818631 (+ 1421us) server_negotiation.cc:1035] Waiting for connection context
0926 03:47:13.818701 (+ 70us) server_negotiation.cc:299] Negotiation successful
0926 03:47:13.818895 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14952,"thread_start_us":14800,"threads_started":1}
I20260926 03:47:13.820092 14919 tablet_service.cc:1467] Processing CreateTablet for tablet 4efa8d70e99d455890526935170f556f (DEFAULT_TABLE table=test-table [id=609201833874441880b3872669cdc56a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:47:13.820420 14785 tablet_service.cc:1467] Processing CreateTablet for tablet 4efa8d70e99d455890526935170f556f (DEFAULT_TABLE table=test-table [id=609201833874441880b3872669cdc56a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:47:13.821430 14919 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4efa8d70e99d455890526935170f556f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:47:13.821957 14785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4efa8d70e99d455890526935170f556f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:47:13.822716 15053 tablet_service.cc:1467] Processing CreateTablet for tablet 4efa8d70e99d455890526935170f556f (DEFAULT_TABLE table=test-table [id=609201833874441880b3872669cdc56a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:47:13.824141 15053 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4efa8d70e99d455890526935170f556f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:47:13.833325 15137 tablet_bootstrap.cc:492] T 4efa8d70e99d455890526935170f556f P ca26e0bdcc7c4021a1ae2b833aa2695f: Bootstrap starting.
I20260926 03:47:13.837271 15137 tablet_bootstrap.cc:654] T 4efa8d70e99d455890526935170f556f P ca26e0bdcc7c4021a1ae2b833aa2695f: Neither blocks nor log segments found. Creating new log.
I20260926 03:47:13.838124 15138 tablet_bootstrap.cc:492] T 4efa8d70e99d455890526935170f556f P 26e7194486b140e2b7434d73bbe399e6: Bootstrap starting.
I20260926 03:47:13.838928 15137 log.cc:824] T 4efa8d70e99d455890526935170f556f P ca26e0bdcc7c4021a1ae2b833aa2695f: Log is configured to *not* fsync() on all Append() calls
I20260926 03:47:13.842435 15137 tablet_bootstrap.cc:492] T 4efa8d70e99d455890526935170f556f P ca26e0bdcc7c4021a1ae2b833aa2695f: No bootstrap required, opened a new log
I20260926 03:47:13.842453 15138 tablet_bootstrap.cc:654] T 4efa8d70e99d455890526935170f556f P 26e7194486b140e2b7434d73bbe399e6: Neither blocks nor log segments found. Creating new log.
I20260926 03:47:13.842695 15139 tablet_bootstrap.cc:492] T 4efa8d70e99d455890526935170f556f P 453de609dea34f9eb8f70055f5398b7e: Bootstrap starting.
I20260926 03:47:13.842769 15137 ts_tablet_manager.cc:1397] T 4efa8d70e99d455890526935170f556f P ca26e0bdcc7c4021a1ae2b833aa2695f: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260926 03:47:13.843791 15138 log.cc:824] T 4efa8d70e99d455890526935170f556f P 26e7194486b140e2b7434d73bbe399e6: Log is configured to *not* fsync() on all Append() calls
I20260926 03:47:13.846180 15139 tablet_bootstrap.cc:654] T 4efa8d70e99d455890526935170f556f P 453de609dea34f9eb8f70055f5398b7e: Neither blocks nor log segments found. Creating new log.
I20260926 03:47:13.847404 15138 tablet_bootstrap.cc:492] T 4efa8d70e99d455890526935170f556f P 26e7194486b140e2b7434d73bbe399e6: No bootstrap required, opened a new log
I20260926 03:47:13.847599 15139 log.cc:824] T 4efa8d70e99d455890526935170f556f P 453de609dea34f9eb8f70055f5398b7e: Log is configured to *not* fsync() on all Append() calls
I20260926 03:47:13.847702 15138 ts_tablet_manager.cc:1397] T 4efa8d70e99d455890526935170f556f P 26e7194486b140e2b7434d73bbe399e6: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.005s
I20260926 03:47:13.851043 15139 tablet_bootstrap.cc:492] T 4efa8d70e99d455890526935170f556f P 453de609dea34f9eb8f70055f5398b7e: No bootstrap required, opened a new log
I20260926 03:47:13.851343 15139 ts_tablet_manager.cc:1397] T 4efa8d70e99d455890526935170f556f P 453de609dea34f9eb8f70055f5398b7e: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260926 03:47:13.851924 15137 raft_consensus.cc:348] T 4efa8d70e99d455890526935170f556f P ca26e0bdcc7c4021a1ae2b833aa2695f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca26e0bdcc7c4021a1ae2b833aa2695f" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 34653 } } peers { permanent_uuid: "26e7194486b140e2b7434d73bbe399e6" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 38899 } } peers { permanent_uuid: "453de609dea34f9eb8f70055f5398b7e" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 33219 } }
I20260926 03:47:13.852350 15137 raft_consensus.cc:374] T 4efa8d70e99d455890526935170f556f P ca26e0bdcc7c4021a1ae2b833aa2695f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:47:13.852447 15137 raft_consensus.cc:729] T 4efa8d70e99d455890526935170f556f P ca26e0bdcc7c4021a1ae2b833aa2695f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca26e0bdcc7c4021a1ae2b833aa2695f, State: Initialized, Role: FOLLOWER
I20260926 03:47:13.853112 15137 consensus_queue.cc:260] T 4efa8d70e99d455890526935170f556f P ca26e0bdcc7c4021a1ae2b833aa2695f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca26e0bdcc7c4021a1ae2b833aa2695f" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 34653 } } peers { permanent_uuid: "26e7194486b140e2b7434d73bbe399e6" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 38899 } } peers { permanent_uuid: "453de609dea34f9eb8f70055f5398b7e" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 33219 } }
I20260926 03:47:13.854270 14984 heartbeater.cc:503] Master 127.0.194.254:34561 was elected leader, sending a full tablet report...
I20260926 03:47:13.855005 15137 ts_tablet_manager.cc:1428] T 4efa8d70e99d455890526935170f556f P ca26e0bdcc7c4021a1ae2b833aa2695f: Time spent starting tablet: real 0.012s user 0.003s sys 0.009s
I20260926 03:47:13.858417 15138 raft_consensus.cc:348] T 4efa8d70e99d455890526935170f556f P 26e7194486b140e2b7434d73bbe399e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca26e0bdcc7c4021a1ae2b833aa2695f" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 34653 } } peers { permanent_uuid: "26e7194486b140e2b7434d73bbe399e6" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 38899 } } peers { permanent_uuid: "453de609dea34f9eb8f70055f5398b7e" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 33219 } }
I20260926 03:47:13.858929 15138 raft_consensus.cc:374] T 4efa8d70e99d455890526935170f556f P 26e7194486b140e2b7434d73bbe399e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:47:13.859035 15138 raft_consensus.cc:729] T 4efa8d70e99d455890526935170f556f P 26e7194486b140e2b7434d73bbe399e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26e7194486b140e2b7434d73bbe399e6, State: Initialized, Role: FOLLOWER
I20260926 03:47:13.859735 15138 consensus_queue.cc:260] T 4efa8d70e99d455890526935170f556f P 26e7194486b140e2b7434d73bbe399e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca26e0bdcc7c4021a1ae2b833aa2695f" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 34653 } } peers { permanent_uuid: "26e7194486b140e2b7434d73bbe399e6" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 38899 } } peers { permanent_uuid: "453de609dea34f9eb8f70055f5398b7e" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 33219 } }
I20260926 03:47:13.861613 15138 ts_tablet_manager.cc:1428] T 4efa8d70e99d455890526935170f556f P 26e7194486b140e2b7434d73bbe399e6: Time spent starting tablet: real 0.014s user 0.007s sys 0.006s
I20260926 03:47:13.862857 15139 raft_consensus.cc:348] T 4efa8d70e99d455890526935170f556f P 453de609dea34f9eb8f70055f5398b7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca26e0bdcc7c4021a1ae2b833aa2695f" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 34653 } } peers { permanent_uuid: "26e7194486b140e2b7434d73bbe399e6" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 38899 } } peers { permanent_uuid: "453de609dea34f9eb8f70055f5398b7e" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 33219 } }
I20260926 03:47:13.863286 15139 raft_consensus.cc:374] T 4efa8d70e99d455890526935170f556f P 453de609dea34f9eb8f70055f5398b7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:47:13.863379 15139 raft_consensus.cc:729] T 4efa8d70e99d455890526935170f556f P 453de609dea34f9eb8f70055f5398b7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 453de609dea34f9eb8f70055f5398b7e, State: Initialized, Role: FOLLOWER
I20260926 03:47:13.863947 15139 consensus_queue.cc:260] T 4efa8d70e99d455890526935170f556f P 453de609dea34f9eb8f70055f5398b7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca26e0bdcc7c4021a1ae2b833aa2695f" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 34653 } } peers { permanent_uuid: "26e7194486b140e2b7434d73bbe399e6" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 38899 } } peers { permanent_uuid: "453de609dea34f9eb8f70055f5398b7e" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 33219 } }
I20260926 03:47:13.865137 15118 heartbeater.cc:503] Master 127.0.194.254:34561 was elected leader, sending a full tablet report...
I20260926 03:47:13.865744 15139 ts_tablet_manager.cc:1428] T 4efa8d70e99d455890526935170f556f P 453de609dea34f9eb8f70055f5398b7e: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260926 03:47:13.888007 15144 raft_consensus.cc:482] T 4efa8d70e99d455890526935170f556f P 26e7194486b140e2b7434d73bbe399e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 03:47:13.888271 15144 raft_consensus.cc:504] T 4efa8d70e99d455890526935170f556f P 26e7194486b140e2b7434d73bbe399e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca26e0bdcc7c4021a1ae2b833aa2695f" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 34653 } } peers { permanent_uuid: "26e7194486b140e2b7434d73bbe399e6" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 38899 } } peers { permanent_uuid: "453de609dea34f9eb8f70055f5398b7e" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 33219 } }
I20260926 03:47:13.889997 15144 leader_election.cc:288] T 4efa8d70e99d455890526935170f556f P 26e7194486b140e2b7434d73bbe399e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ca26e0bdcc7c4021a1ae2b833aa2695f (127.0.194.194:34653), 453de609dea34f9eb8f70055f5398b7e (127.0.194.195:33219)
I20260926 03:47:13.899121 15136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:13.890632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.193:37527 (local address 127.0.194.195:33219)
0926 03:47:13.890901 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0926 03:47:13.890909 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:47:13.891503 (+ 594us) server_negotiation.cc:407] Connection header received
0926 03:47:13.891615 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:47:13.891620 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:47:13.891693 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:47:13.891780 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:47:13.892553 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:47:13.893680 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:47:13.896708 (+ 3028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:47:13.897945 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:47:13.898111 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:47:13.898188 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 03:47:13.898697 (+ 509us) server_negotiation.cc:299] Negotiation successful
0926 03:47:13.898852 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260926 03:47:13.899122 15133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:13.890314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.193:41401 (local address 127.0.194.194:34653)
0926 03:47:13.890559 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0926 03:47:13.890568 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:47:13.891338 (+ 770us) server_negotiation.cc:407] Connection header received
0926 03:47:13.891559 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:47:13.891565 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:47:13.891645 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:47:13.891753 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:47:13.892545 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:47:13.893667 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:47:13.896700 (+ 3033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:47:13.897945 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:47:13.898116 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:47:13.898188 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 03:47:13.898697 (+ 509us) server_negotiation.cc:299] Negotiation successful
0926 03:47:13.898852 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 03:47:13.899506 15147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:13.890219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.194:34653 (local address 127.0.194.193:41401)
0926 03:47:13.891198 (+ 979us) negotiation.cc:107] Waiting for socket to connect
0926 03:47:13.891226 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0926 03:47:13.891396 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:47:13.891810 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:47:13.891815 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:47:13.891849 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:47:13.892400 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:47:13.892409 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:47:13.893818 (+ 1409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:47:13.893825 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:47:13.896539 (+ 2714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:47:13.896552 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:47:13.898252 (+ 1700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:47:13.898261 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:47:13.898495 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:47:13.898549 (+ 54us) client_negotiation.cc:769] Sending connection context
0926 03:47:13.898661 (+ 112us) client_negotiation.cc:240] Negotiation successful
0926 03:47:13.898949 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":809,"mutex_wait_us":353,"thread_start_us":882,"threads_started":1}
I20260926 03:47:13.899469 15146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:13.890253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.195:33219 (local address 127.0.194.193:37527)
0926 03:47:13.891394 (+ 1141us) negotiation.cc:107] Waiting for socket to connect
0926 03:47:13.891409 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 03:47:13.891515 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:47:13.891815 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:47:13.891818 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:47:13.891921 (+ 103us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:47:13.892376 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:47:13.892396 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:47:13.893818 (+ 1422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:47:13.893825 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:47:13.896535 (+ 2710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:47:13.896552 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:47:13.898270 (+ 1718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:47:13.898276 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:47:13.898491 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:47:13.898556 (+ 65us) client_negotiation.cc:769] Sending connection context
0926 03:47:13.898661 (+ 105us) client_negotiation.cc:240] Negotiation successful
0926 03:47:13.898944 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1030,"mutex_wait_us":208,"spinlock_wait_cycles":25472,"thread_start_us":537,"threads_started":1}
I20260926 03:47:13.900595 14939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4efa8d70e99d455890526935170f556f" candidate_uuid: "26e7194486b140e2b7434d73bbe399e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca26e0bdcc7c4021a1ae2b833aa2695f" is_pre_election: true
I20260926 03:47:13.900620 15073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4efa8d70e99d455890526935170f556f" candidate_uuid: "26e7194486b140e2b7434d73bbe399e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "453de609dea34f9eb8f70055f5398b7e" is_pre_election: true
I20260926 03:47:13.901137 14939 raft_consensus.cc:2454] T 4efa8d70e99d455890526935170f556f P ca26e0bdcc7c4021a1ae2b833aa2695f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26e7194486b140e2b7434d73bbe399e6 in term 0.
I20260926 03:47:13.901224 15073 raft_consensus.cc:2454] T 4efa8d70e99d455890526935170f556f P 453de609dea34f9eb8f70055f5398b7e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26e7194486b140e2b7434d73bbe399e6 in term 0.
I20260926 03:47:13.901957 14741 leader_election.cc:302] T 4efa8d70e99d455890526935170f556f P 26e7194486b140e2b7434d73bbe399e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 26e7194486b140e2b7434d73bbe399e6, 453de609dea34f9eb8f70055f5398b7e, ca26e0bdcc7c4021a1ae2b833aa2695f; no voters:
I20260926 03:47:13.902432 15144 raft_consensus.cc:2781] T 4efa8d70e99d455890526935170f556f P 26e7194486b140e2b7434d73bbe399e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 03:47:13.902601 15144 raft_consensus.cc:482] T 4efa8d70e99d455890526935170f556f P 26e7194486b140e2b7434d73bbe399e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 03:47:13.902761 15144 raft_consensus.cc:3037] T 4efa8d70e99d455890526935170f556f P 26e7194486b140e2b7434d73bbe399e6 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:47:13.904930 15144 raft_consensus.cc:504] T 4efa8d70e99d455890526935170f556f P 26e7194486b140e2b7434d73bbe399e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca26e0bdcc7c4021a1ae2b833aa2695f" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 34653 } } peers { permanent_uuid: "26e7194486b140e2b7434d73bbe399e6" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 38899 } } peers { permanent_uuid: "453de609dea34f9eb8f70055f5398b7e" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 33219 } }
I20260926 03:47:13.905575 15144 leader_election.cc:288] T 4efa8d70e99d455890526935170f556f P 26e7194486b140e2b7434d73bbe399e6 [CANDIDATE]: Term 1 election: Requested vote from peers ca26e0bdcc7c4021a1ae2b833aa2695f (127.0.194.194:34653), 453de609dea34f9eb8f70055f5398b7e (127.0.194.195:33219)
I20260926 03:47:13.905995 14939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4efa8d70e99d455890526935170f556f" candidate_uuid: "26e7194486b140e2b7434d73bbe399e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca26e0bdcc7c4021a1ae2b833aa2695f"
I20260926 03:47:13.906236 14939 raft_consensus.cc:3037] T 4efa8d70e99d455890526935170f556f P ca26e0bdcc7c4021a1ae2b833aa2695f [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:47:13.906199 15073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4efa8d70e99d455890526935170f556f" candidate_uuid: "26e7194486b140e2b7434d73bbe399e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "453de609dea34f9eb8f70055f5398b7e"
I20260926 03:47:13.906427 15073 raft_consensus.cc:3037] T 4efa8d70e99d455890526935170f556f P 453de609dea34f9eb8f70055f5398b7e [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:47:13.908601 14939 raft_consensus.cc:2454] T 4efa8d70e99d455890526935170f556f P ca26e0bdcc7c4021a1ae2b833aa2695f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26e7194486b140e2b7434d73bbe399e6 in term 1.
I20260926 03:47:13.908723 15073 raft_consensus.cc:2454] T 4efa8d70e99d455890526935170f556f P 453de609dea34f9eb8f70055f5398b7e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26e7194486b140e2b7434d73bbe399e6 in term 1.
I20260926 03:47:13.909057 14738 leader_election.cc:302] T 4efa8d70e99d455890526935170f556f P 26e7194486b140e2b7434d73bbe399e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26e7194486b140e2b7434d73bbe399e6, ca26e0bdcc7c4021a1ae2b833aa2695f; no voters:
I20260926 03:47:13.909415 15144 raft_consensus.cc:2781] T 4efa8d70e99d455890526935170f556f P 26e7194486b140e2b7434d73bbe399e6 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:47:13.909893 15144 raft_consensus.cc:686] T 4efa8d70e99d455890526935170f556f P 26e7194486b140e2b7434d73bbe399e6 [term 1 LEADER]: Becoming Leader. State: Replica: 26e7194486b140e2b7434d73bbe399e6, State: Running, Role: LEADER
I20260926 03:47:13.910449 15144 consensus_queue.cc:237] T 4efa8d70e99d455890526935170f556f P 26e7194486b140e2b7434d73bbe399e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca26e0bdcc7c4021a1ae2b833aa2695f" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 34653 } } peers { permanent_uuid: "26e7194486b140e2b7434d73bbe399e6" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 38899 } } peers { permanent_uuid: "453de609dea34f9eb8f70055f5398b7e" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 33219 } }
W20260926 03:47:13.914995 14851 tablet.cc:2369] T 4efa8d70e99d455890526935170f556f P 26e7194486b140e2b7434d73bbe399e6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 03:47:13.915585 14661 catalog_manager.cc:5495] T 4efa8d70e99d455890526935170f556f P 26e7194486b140e2b7434d73bbe399e6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 26e7194486b140e2b7434d73bbe399e6 (127.0.194.193). New cstate: current_term: 1 leader_uuid: "26e7194486b140e2b7434d73bbe399e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca26e0bdcc7c4021a1ae2b833aa2695f" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 34653 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26e7194486b140e2b7434d73bbe399e6" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 38899 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "453de609dea34f9eb8f70055f5398b7e" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 33219 } health_report { overall_health: UNKNOWN } } }
W20260926 03:47:13.939522 14985 tablet.cc:2369] T 4efa8d70e99d455890526935170f556f P ca26e0bdcc7c4021a1ae2b833aa2695f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 03:47:13.942898 15134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:13.937219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50278 (local address 127.0.194.193:38899)
0926 03:47:13.937443 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0926 03:47:13.937448 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:47:13.937469 (+ 21us) server_negotiation.cc:407] Connection header received
0926 03:47:13.937652 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:47:13.937656 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:47:13.937732 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:47:13.937867 (+ 135us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:47:13.938197 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:47:13.939001 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:47:13.940830 (+ 1829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:47:13.941240 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:47:13.941337 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:47:13.941810 (+ 473us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:47:13.942302 (+ 492us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:47:13.942416 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0926 03:47:13.942496 (+ 80us) server_negotiation.cc:299] Negotiation successful
0926 03:47:13.942560 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260926 03:47:13.951169 15073 raft_consensus.cc:1261] T 4efa8d70e99d455890526935170f556f P 453de609dea34f9eb8f70055f5398b7e [term 1 FOLLOWER]: Refusing update from remote peer 26e7194486b140e2b7434d73bbe399e6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:47:13.951169 14939 raft_consensus.cc:1261] T 4efa8d70e99d455890526935170f556f P ca26e0bdcc7c4021a1ae2b833aa2695f [term 1 FOLLOWER]: Refusing update from remote peer 26e7194486b140e2b7434d73bbe399e6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:47:13.952226 15148 consensus_queue.cc:1035] T 4efa8d70e99d455890526935170f556f P 26e7194486b140e2b7434d73bbe399e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453de609dea34f9eb8f70055f5398b7e" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 33219 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:47:13.952581 15144 consensus_queue.cc:1035] T 4efa8d70e99d455890526935170f556f P 26e7194486b140e2b7434d73bbe399e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca26e0bdcc7c4021a1ae2b833aa2695f" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 34653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:47:13.965204 15153 mvcc.cc:205] Tried to move back new op lower bound from 7333455601453260800 to 7333455601301299200. Current Snapshot: MvccSnapshot[applied={T|T < 7333455601453260800}]
I20260926 03:47:13.966076 15152 mvcc.cc:205] Tried to move back new op lower bound from 7333455601453260800 to 7333455601301299200. Current Snapshot: MvccSnapshot[applied={T|T < 7333455601453260800}]
I20260926 03:47:13.982450 14660 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59628:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:47:13.982838 14660 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59628:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:47:13.987053 14660 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fd19af12fdda4aa9be763b660e4611c8: Sending DeleteTablet for 3 replicas of tablet 4efa8d70e99d455890526935170f556f
I20260926 03:47:13.988292 15053 tablet_service.cc:1514] Processing DeleteTablet for tablet 4efa8d70e99d455890526935170f556f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:47:13 UTC) from {username='kudu', principal='kudu/127.0.194.254@KRBTEST.COM'} at 127.0.0.1:37450
I20260926 03:47:13.988296 14919 tablet_service.cc:1514] Processing DeleteTablet for tablet 4efa8d70e99d455890526935170f556f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:47:13 UTC) from {username='kudu', principal='kudu/127.0.194.254@KRBTEST.COM'} at 127.0.0.1:40054
I20260926 03:47:13.989120 14785 tablet_service.cc:1514] Processing DeleteTablet for tablet 4efa8d70e99d455890526935170f556f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:47:13 UTC) from {username='kudu', principal='kudu/127.0.194.254@KRBTEST.COM'} at 127.0.0.1:50274
I20260926 03:47:13.989851 15162 tablet_replica.cc:330] T 4efa8d70e99d455890526935170f556f P 453de609dea34f9eb8f70055f5398b7e: stopping tablet replica
I20260926 03:47:13.990363 15163 tablet_replica.cc:330] T 4efa8d70e99d455890526935170f556f P 26e7194486b140e2b7434d73bbe399e6: stopping tablet replica
I20260926 03:47:13.989261 15161 tablet_replica.cc:330] T 4efa8d70e99d455890526935170f556f P ca26e0bdcc7c4021a1ae2b833aa2695f: stopping tablet replica
I20260926 03:47:13.990640 779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu pid 14722
I20260926 03:47:13.990847 15163 raft_consensus.cc:2229] T 4efa8d70e99d455890526935170f556f P 26e7194486b140e2b7434d73bbe399e6 [term 1 LEADER]: Raft consensus shutting down.
I20260926 03:47:13.991082 15161 raft_consensus.cc:2229] T 4efa8d70e99d455890526935170f556f P ca26e0bdcc7c4021a1ae2b833aa2695f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:47:13.990350 15162 raft_consensus.cc:2229] T 4efa8d70e99d455890526935170f556f P 453de609dea34f9eb8f70055f5398b7e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:47:13.991744 15161 raft_consensus.cc:2258] T 4efa8d70e99d455890526935170f556f P ca26e0bdcc7c4021a1ae2b833aa2695f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:47:13.991963 15163 raft_consensus.cc:2258] T 4efa8d70e99d455890526935170f556f P 26e7194486b140e2b7434d73bbe399e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:47:13.992405 15162 raft_consensus.cc:2258] T 4efa8d70e99d455890526935170f556f P 453de609dea34f9eb8f70055f5398b7e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:47:13.994339 15161 ts_tablet_manager.cc:1905] T 4efa8d70e99d455890526935170f556f P ca26e0bdcc7c4021a1ae2b833aa2695f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:47:13.994355 15163 ts_tablet_manager.cc:1905] T 4efa8d70e99d455890526935170f556f P 26e7194486b140e2b7434d73bbe399e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:47:13.999286 15162 ts_tablet_manager.cc:1905] T 4efa8d70e99d455890526935170f556f P 453de609dea34f9eb8f70055f5398b7e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:47:14.001663 15161 ts_tablet_manager.cc:1918] T 4efa8d70e99d455890526935170f556f P ca26e0bdcc7c4021a1ae2b833aa2695f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:47:14.001890 15161 log.cc:1196] T 4efa8d70e99d455890526935170f556f P ca26e0bdcc7c4021a1ae2b833aa2695f: Deleting WAL directory at /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/minicluster-data/ts-1/wal/wals/4efa8d70e99d455890526935170f556f
I20260926 03:47:14.002610 15161 ts_tablet_manager.cc:1939] T 4efa8d70e99d455890526935170f556f P ca26e0bdcc7c4021a1ae2b833aa2695f: Deleting consensus metadata
I20260926 03:47:14.003285 15163 ts_tablet_manager.cc:1918] T 4efa8d70e99d455890526935170f556f P 26e7194486b140e2b7434d73bbe399e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:47:14.003494 15163 log.cc:1196] T 4efa8d70e99d455890526935170f556f P 26e7194486b140e2b7434d73bbe399e6: Deleting WAL directory at /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/minicluster-data/ts-0/wal/wals/4efa8d70e99d455890526935170f556f
I20260926 03:47:14.004076 15163 ts_tablet_manager.cc:1939] T 4efa8d70e99d455890526935170f556f P 26e7194486b140e2b7434d73bbe399e6: Deleting consensus metadata
I20260926 03:47:14.004758 14645 catalog_manager.cc:4841] TS ca26e0bdcc7c4021a1ae2b833aa2695f (127.0.194.194:34653): tablet 4efa8d70e99d455890526935170f556f (table test-table [id=609201833874441880b3872669cdc56a]) successfully deleted
Sep 26 03:47:14 dist-test-slave-n2th krb5kdc[14616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394431, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.194.193@KRBTEST.COM
I20260926 03:47:14.005853 14647 catalog_manager.cc:4841] TS 26e7194486b140e2b7434d73bbe399e6 (127.0.194.193:38899): tablet 4efa8d70e99d455890526935170f556f (table test-table [id=609201833874441880b3872669cdc56a]) successfully deleted
I20260926 03:47:14.006531 15162 ts_tablet_manager.cc:1918] T 4efa8d70e99d455890526935170f556f P 453de609dea34f9eb8f70055f5398b7e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:47:14.006708 15162 log.cc:1196] T 4efa8d70e99d455890526935170f556f P 453de609dea34f9eb8f70055f5398b7e: Deleting WAL directory at /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/minicluster-data/ts-2/wal/wals/4efa8d70e99d455890526935170f556f
I20260926 03:47:14.007179 15162 ts_tablet_manager.cc:1939] T 4efa8d70e99d455890526935170f556f P 453de609dea34f9eb8f70055f5398b7e: Deleting consensus metadata
I20260926 03:47:14.008824 14648 catalog_manager.cc:4841] TS 453de609dea34f9eb8f70055f5398b7e (127.0.194.195:33219): tablet 4efa8d70e99d455890526935170f556f (table test-table [id=609201833874441880b3872669cdc56a]) successfully deleted
I20260926 03:47:14.014678 15134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:13.992760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50280 (local address 127.0.194.193:38899)
0926 03:47:13.992939 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0926 03:47:13.992945 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:47:13.993503 (+ 558us) server_negotiation.cc:407] Connection header received
0926 03:47:13.993802 (+ 299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:47:13.993808 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:47:13.993904 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:47:13.993981 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:47:13.996169 (+ 2188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:47:13.997395 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:47:13.999424 (+ 2029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:47:14.000163 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:47:14.000273 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:47:14.008116 (+ 7843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:47:14.008130 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:47:14.008148 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:47:14.008203 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:47:14.011590 (+ 3387us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:47:14.012534 (+ 944us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:47:14.012540 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:47:14.012543 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:47:14.012686 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:47:14.013139 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:47:14.013145 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:47:14.013151 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:47:14.013584 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:47:14.013822 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0926 03:47:14.014217 (+ 395us) server_negotiation.cc:299] Negotiation successful
0926 03:47:14.014395 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 03:47:14.015595 14845 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:47:14.127874 779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu with pid 14722
I20260926 03:47:14.147012 779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu pid 14856
Sep 26 03:47:14 dist-test-slave-n2th krb5kdc[14616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394431, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.194.194@KRBTEST.COM
I20260926 03:47:14.163439 15133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:14.147771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40070 (local address 127.0.194.194:34653)
0926 03:47:14.147955 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0926 03:47:14.147963 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:47:14.147985 (+ 22us) server_negotiation.cc:407] Connection header received
0926 03:47:14.148192 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:47:14.148201 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:47:14.148263 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:47:14.148383 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:47:14.149989 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:47:14.150957 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:47:14.152197 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:47:14.152666 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:47:14.152806 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:47:14.157597 (+ 4791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:47:14.157613 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:47:14.157656 (+ 43us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:47:14.157695 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:47:14.160501 (+ 2806us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:47:14.161260 (+ 759us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:47:14.161268 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:47:14.161271 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:47:14.161393 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:47:14.161775 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:47:14.161780 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:47:14.161783 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:47:14.162176 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:47:14.162476 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0926 03:47:14.162858 (+ 382us) server_negotiation.cc:299] Negotiation successful
0926 03:47:14.163088 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 03:47:14.164242 14979 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:47:14.254786 779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu with pid 14856
I20260926 03:47:14.272042 779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu pid 14990
Sep 26 03:47:14 dist-test-slave-n2th krb5kdc[14616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394431, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.194.195@KRBTEST.COM
I20260926 03:47:14.288587 15136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:14.272849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37458 (local address 127.0.194.195:33219)
0926 03:47:14.273062 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0926 03:47:14.273068 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:47:14.273089 (+ 21us) server_negotiation.cc:407] Connection header received
0926 03:47:14.273297 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:47:14.273302 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:47:14.273381 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:47:14.273492 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:47:14.274927 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:47:14.275661 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:47:14.276945 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:47:14.277355 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:47:14.277503 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:47:14.282354 (+ 4851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:47:14.282376 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:47:14.282393 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:47:14.282435 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:47:14.285385 (+ 2950us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:47:14.286263 (+ 878us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:47:14.286271 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:47:14.286274 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:47:14.286439 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:47:14.286914 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:47:14.286919 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:47:14.286933 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:47:14.287322 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:47:14.287632 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0926 03:47:14.288007 (+ 375us) server_negotiation.cc:299] Negotiation successful
0926 03:47:14.288259 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 03:47:14.289372 15113 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:47:14.386893 779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu with pid 14990
I20260926 03:47:14.403764 779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu pid 14629
I20260926 03:47:14.404872 14691 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:47:14.519325 779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu with pid 14629
I20260926 03:47:14.540935 779 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
/tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.194.254:34561
--webserver_interface=127.0.194.254
--webserver_port=37679
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/krb5kdc/kudu_127.0.194.254.keytab
--principal=kudu/127.0.194.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.194.212:45229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.194.254:34561
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:47:14.879801 15169 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:47:14.880182 15169 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:47:14.880345 15169 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:47:14.887902 15169 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:47:14.888062 15169 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:47:14.888182 15169 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:47:14.888281 15169 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:47:14.888366 15169 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:47:14.899418 15169 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.194.212:45229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.0.194.254:34561
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/krb5kdc/kudu_127.0.194.254.keytab
--principal=kudu/127.0.194.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.194.254:34561
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.194.254
--webserver_port=37679
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:47:14.900830 15169 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:47:14.902257 15169 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:47:14.925668 15175 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:47:14.925612 15176 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:47:14.925606 15178 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:47:14.926596 15169 server_base.cc:1034] running on GCE node
I20260926 03:47:14.927608 15169 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:47:14.929303 15169 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:47:14.930541 15169 hybrid_clock.cc:648] HybridClock initialized: now 1790394434930489 us; error 47 us; skew 500 ppm
I20260926 03:47:14.931025 15169 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:47:14 dist-test-slave-n2th krb5kdc[14616](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394434, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:47:14.934618 15169 init.cc:377] Logged in from keytab as kudu/127.0.194.254@KRBTEST.COM (short username kudu)
I20260926 03:47:14.936362 15169 webserver.cc:460] Webserver started at http://127.0.194.254:37679/ using document root <none> and password file <none>
I20260926 03:47:14.937158 15169 fs_manager.cc:359] Metadata directory not provided
I20260926 03:47:14.937338 15169 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:47:14.943817 15169 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 03:47:14.947067 15185 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:47:14.948539 15169 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 03:47:14.948748 15169 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/minicluster-data/master-0/data,/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/minicluster-data/master-0/wal
uuid: "fd19af12fdda4aa9be763b660e4611c8"
format_stamp: "Formatted at 2026-09-26 03:47:11 on dist-test-slave-n2th"
server_key: "a8703d5d0686fe8f2af4d77f386693cf"
server_key_iv: "571485da51254437ea71931d7e4ab0d3"
server_key_version: "encryptionkey@0"
I20260926 03:47:14.949903 15169 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:47:14.972798 15169 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:47:14.977252 15169 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:47:14.977561 15169 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:47:14.997978 15169 rpc_server.cc:307] RPC server started. Bound to: 127.0.194.254:34561
I20260926 03:47:14.998013 15236 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.194.254:34561 every 8 connection(s)
I20260926 03:47:14.999677 15169 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/minicluster-data/master-0/data/info.pb
I20260926 03:47:15.005430 779 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu as pid 15169
I20260926 03:47:15.006263 779 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
/tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.194.193:38899
--local_ip_for_outbound_sockets=127.0.194.193
--tserver_master_addrs=127.0.194.254:34561
--webserver_port=45389
--webserver_interface=127.0.194.193
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/krb5kdc/kudu_127.0.194.193.keytab
--principal=kudu/127.0.194.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.194.212:45229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 03:47:15.007493 15237 sys_catalog.cc:263] Verifying existing consensus state
I20260926 03:47:15.011113 15237 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd19af12fdda4aa9be763b660e4611c8: Bootstrap starting.
I20260926 03:47:15.040180 15237 log.cc:824] T 00000000000000000000000000000000 P fd19af12fdda4aa9be763b660e4611c8: Log is configured to *not* fsync() on all Append() calls
I20260926 03:47:15.056831 15237 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd19af12fdda4aa9be763b660e4611c8: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 03:47:15.057538 15237 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd19af12fdda4aa9be763b660e4611c8: Bootstrap complete.
I20260926 03:47:15.071943 15237 raft_consensus.cc:348] T 00000000000000000000000000000000 P fd19af12fdda4aa9be763b660e4611c8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd19af12fdda4aa9be763b660e4611c8" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 34561 } }
I20260926 03:47:15.073211 15237 raft_consensus.cc:729] T 00000000000000000000000000000000 P fd19af12fdda4aa9be763b660e4611c8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd19af12fdda4aa9be763b660e4611c8, State: Initialized, Role: FOLLOWER
I20260926 03:47:15.074080 15237 consensus_queue.cc:260] T 00000000000000000000000000000000 P fd19af12fdda4aa9be763b660e4611c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd19af12fdda4aa9be763b660e4611c8" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 34561 } }
I20260926 03:47:15.074450 15237 raft_consensus.cc:388] T 00000000000000000000000000000000 P fd19af12fdda4aa9be763b660e4611c8 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:47:15.074616 15237 raft_consensus.cc:482] T 00000000000000000000000000000000 P fd19af12fdda4aa9be763b660e4611c8 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:47:15.074887 15237 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fd19af12fdda4aa9be763b660e4611c8 [term 1 FOLLOWER]: Advancing to term 2
I20260926 03:47:15.081641 15237 raft_consensus.cc:504] T 00000000000000000000000000000000 P fd19af12fdda4aa9be763b660e4611c8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd19af12fdda4aa9be763b660e4611c8" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 34561 } }
I20260926 03:47:15.082365 15237 leader_election.cc:302] T 00000000000000000000000000000000 P fd19af12fdda4aa9be763b660e4611c8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fd19af12fdda4aa9be763b660e4611c8; no voters:
I20260926 03:47:15.083103 15237 leader_election.cc:288] T 00000000000000000000000000000000 P fd19af12fdda4aa9be763b660e4611c8 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 03:47:15.083292 15241 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fd19af12fdda4aa9be763b660e4611c8 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 03:47:15.084584 15241 raft_consensus.cc:686] T 00000000000000000000000000000000 P fd19af12fdda4aa9be763b660e4611c8 [term 2 LEADER]: Becoming Leader. State: Replica: fd19af12fdda4aa9be763b660e4611c8, State: Running, Role: LEADER
I20260926 03:47:15.085126 15241 consensus_queue.cc:237] T 00000000000000000000000000000000 P fd19af12fdda4aa9be763b660e4611c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd19af12fdda4aa9be763b660e4611c8" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 34561 } }
I20260926 03:47:15.085418 15237 sys_catalog.cc:564] T 00000000000000000000000000000000 P fd19af12fdda4aa9be763b660e4611c8 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:47:15.087744 15243 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd19af12fdda4aa9be763b660e4611c8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fd19af12fdda4aa9be763b660e4611c8. Latest consensus state: current_term: 2 leader_uuid: "fd19af12fdda4aa9be763b660e4611c8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd19af12fdda4aa9be763b660e4611c8" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 34561 } } }
I20260926 03:47:15.088130 15243 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd19af12fdda4aa9be763b660e4611c8 [sys.catalog]: This master's current role is: LEADER
I20260926 03:47:15.088685 15242 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd19af12fdda4aa9be763b660e4611c8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "fd19af12fdda4aa9be763b660e4611c8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd19af12fdda4aa9be763b660e4611c8" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 34561 } } }
I20260926 03:47:15.089042 15242 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd19af12fdda4aa9be763b660e4611c8 [sys.catalog]: This master's current role is: LEADER
I20260926 03:47:15.092705 15247 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:47:15.101608 15247 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:47:15.103530 15247 catalog_manager.cc:1259] Loaded cluster ID: 043e28fe1f644dc78387fad6417f56fa
I20260926 03:47:15.103740 15247 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:47:15.107295 15247 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260926 03:47:15.127635 15247 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260926 03:47:15.137137 15247 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:47:15.139182 15247 catalog_manager.cc:5879] T 00000000000000000000000000000000 P fd19af12fdda4aa9be763b660e4611c8: Loaded TSK: 0
I20260926 03:47:15.140731 15247 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 03:47:15.395089 15239 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:47:15.395411 15239 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:47:15.395582 15239 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:47:15.403422 15239 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:47:15.403556 15239 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:47:15.403745 15239 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.194.193
I20260926 03:47:15.414034 15239 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.194.212:45229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/krb5kdc/kudu_127.0.194.193.keytab
--principal=kudu/127.0.194.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.194.193:38899
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.194.193
--webserver_port=45389
--tserver_master_addrs=127.0.194.254:34561
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.194.193
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:47:15.415459 15239 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:47:15.416816 15239 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:47:15.430800 15265 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:47:15.430754 15266 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:47:15.431588 15268 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:47:15.432178 15239 server_base.cc:1034] running on GCE node
I20260926 03:47:15.432852 15239 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:47:15.434007 15239 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:47:15.435225 15239 hybrid_clock.cc:648] HybridClock initialized: now 1790394435435165 us; error 52 us; skew 500 ppm
I20260926 03:47:15.435562 15239 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:47:15 dist-test-slave-n2th krb5kdc[14616](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394435, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:47:15.443423 15239 init.cc:377] Logged in from keytab as kudu/127.0.194.193@KRBTEST.COM (short username kudu)
I20260926 03:47:15.445245 15239 webserver.cc:460] Webserver started at http://127.0.194.193:45389/ using document root <none> and password file <none>
I20260926 03:47:15.446028 15239 fs_manager.cc:359] Metadata directory not provided
I20260926 03:47:15.446210 15239 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:47:15.452107 15239 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.002s
I20260926 03:47:15.455416 15275 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:47:15.456993 15239 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260926 03:47:15.457206 15239 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/minicluster-data/ts-0/wal
uuid: "26e7194486b140e2b7434d73bbe399e6"
format_stamp: "Formatted at 2026-09-26 03:47:12 on dist-test-slave-n2th"
server_key: "00e3ccacf58a1416f41a36bf1468fda4"
server_key_iv: "68edea37252c95df0e14a621e3c1f2fb"
server_key_version: "encryptionkey@0"
I20260926 03:47:15.458235 15239 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:47:15.477842 15239 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:47:15.482550 15239 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:47:15.482932 15239 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:47:15.484246 15239 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:47:15.485644 15239 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:47:15.485771 15239 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:47:15.485903 15239 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:47:15.485986 15239 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:47:15.520367 15239 rpc_server.cc:307] RPC server started. Bound to: 127.0.194.193:38899
I20260926 03:47:15.520390 15387 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.194.193:38899 every 8 connection(s)
I20260926 03:47:15.521919 15239 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/minicluster-data/ts-0/data/info.pb
I20260926 03:47:15.525902 779 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu as pid 15239
I20260926 03:47:15.526815 779 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
/tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.194.194:34653
--local_ip_for_outbound_sockets=127.0.194.194
--tserver_master_addrs=127.0.194.254:34561
--webserver_port=38561
--webserver_interface=127.0.194.194
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/krb5kdc/kudu_127.0.194.194.keytab
--principal=kudu/127.0.194.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.194.212:45229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 03:47:15 dist-test-slave-n2th krb5kdc[14616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394435, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.193@KRBTEST.COM for kudu/127.0.194.254@KRBTEST.COM
I20260926 03:47:15.548936 15390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:15.525018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.193:44935 (local address 127.0.194.254:34561)
0926 03:47:15.525668 (+ 650us) server_negotiation.cc:206] Beginning negotiation
0926 03:47:15.525687 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 03:47:15.526726 (+ 1039us) server_negotiation.cc:407] Connection header received
0926 03:47:15.529933 (+ 3207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:47:15.529948 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:47:15.530343 (+ 395us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:47:15.530669 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:47:15.532878 (+ 2209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:47:15.534167 (+ 1289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:47:15.535867 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:47:15.536546 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:47:15.536761 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:47:15.540822 (+ 4061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:47:15.540831 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:47:15.540846 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:47:15.540889 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:47:15.544069 (+ 3180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:47:15.545175 (+ 1106us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:47:15.545183 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:47:15.545189 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:47:15.545370 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:47:15.545906 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:47:15.545909 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:47:15.545911 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:47:15.546395 (+ 484us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:47:15.546553 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0926 03:47:15.547637 (+ 1084us) server_negotiation.cc:299] Negotiation successful
0926 03:47:15.547966 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":418,"thread_start_us":249,"threads_started":1}
I20260926 03:47:15.549543 15391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:15.525386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:34561 (local address 127.0.194.193:44935)
0926 03:47:15.526330 (+ 944us) negotiation.cc:107] Waiting for socket to connect
0926 03:47:15.526395 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0926 03:47:15.529261 (+ 2866us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:47:15.530950 (+ 1689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:47:15.530965 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:47:15.531810 (+ 845us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:47:15.532609 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:47:15.532637 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:47:15.534361 (+ 1724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:47:15.534374 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:47:15.535690 (+ 1316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:47:15.535703 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:47:15.536800 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:47:15.536807 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:47:15.537092 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:47:15.537800 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:47:15.537824 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:47:15.540486 (+ 2662us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:47:15.544246 (+ 3760us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:47:15.544258 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:47:15.544289 (+ 31us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:47:15.545017 (+ 728us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:47:15.545491 (+ 474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:47:15.545498 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:47:15.545504 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:47:15.545776 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:47:15.546539 (+ 763us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:47:15.546550 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:47:15.547091 (+ 541us) client_negotiation.cc:769] Sending connection context
0926 03:47:15.547443 (+ 352us) client_negotiation.cc:240] Negotiation successful
0926 03:47:15.548083 (+ 640us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":602,"thread_start_us":372,"threads_started":1}
I20260926 03:47:15.552495 15388 heartbeater.cc:347] Connected to a master server at 127.0.194.254:34561
I20260926 03:47:15.552825 15388 heartbeater.cc:464] Registering TS with master...
I20260926 03:47:15.553733 15388 heartbeater.cc:511] Master 127.0.194.254:34561 requested a full tablet report, sending...
I20260926 03:47:15.556141 15202 ts_manager.cc:194] Registered new tserver with Master: 26e7194486b140e2b7434d73bbe399e6 (127.0.194.193:38899)
I20260926 03:47:15.557634 15202 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.194.193@KRBTEST.COM'} at 127.0.194.193:44935
W20260926 03:47:15.874131 15392 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:47:15.874473 15392 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:47:15.874634 15392 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:47:15.881981 15392 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:47:15.882143 15392 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:47:15.882321 15392 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.194.194
I20260926 03:47:15.893028 15392 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.194.212:45229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/krb5kdc/kudu_127.0.194.194.keytab
--principal=kudu/127.0.194.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.194.194:34653
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.194.194
--webserver_port=38561
--tserver_master_addrs=127.0.194.254:34561
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.194.194
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:47:15.894441 15392 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:47:15.895905 15392 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:47:15.909571 15398 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:47:15.909600 15397 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:47:15.910125 15400 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:47:15.910421 15392 server_base.cc:1034] running on GCE node
I20260926 03:47:15.911135 15392 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:47:15.912154 15392 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:47:15.913383 15392 hybrid_clock.cc:648] HybridClock initialized: now 1790394435913331 us; error 47 us; skew 500 ppm
I20260926 03:47:15.913715 15392 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:47:15 dist-test-slave-n2th krb5kdc[14616](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394435, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:47:15.916981 15392 init.cc:377] Logged in from keytab as kudu/127.0.194.194@KRBTEST.COM (short username kudu)
I20260926 03:47:15.918607 15392 webserver.cc:460] Webserver started at http://127.0.194.194:38561/ using document root <none> and password file <none>
I20260926 03:47:15.919412 15392 fs_manager.cc:359] Metadata directory not provided
I20260926 03:47:15.919574 15392 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:47:15.925493 15392 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 03:47:15.928773 15407 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:47:15.930308 15392 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260926 03:47:15.930521 15392 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/minicluster-data/ts-1/wal
uuid: "ca26e0bdcc7c4021a1ae2b833aa2695f"
format_stamp: "Formatted at 2026-09-26 03:47:13 on dist-test-slave-n2th"
server_key: "df7b7109cc6eb457606f5360d9d6bff3"
server_key_iv: "70ac76747ff308b396c2a02aa945d2ba"
server_key_version: "encryptionkey@0"
I20260926 03:47:15.931636 15392 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:47:15.946336 15392 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:47:15.950958 15392 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:47:15.951248 15392 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:47:15.952293 15392 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:47:15.953763 15392 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:47:15.953881 15392 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:47:15.954003 15392 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:47:15.954093 15392 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:47:15.988287 15392 rpc_server.cc:307] RPC server started. Bound to: 127.0.194.194:34653
I20260926 03:47:15.988384 15519 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.194.194:34653 every 8 connection(s)
I20260926 03:47:15.989847 15392 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/minicluster-data/ts-1/data/info.pb
I20260926 03:47:15.992333 779 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu as pid 15392
I20260926 03:47:15.993054 779 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
/tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.194.195:33219
--local_ip_for_outbound_sockets=127.0.194.195
--tserver_master_addrs=127.0.194.254:34561
--webserver_port=33033
--webserver_interface=127.0.194.195
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/krb5kdc/kudu_127.0.194.195.keytab
--principal=kudu/127.0.194.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.194.212:45229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 03:47:16 dist-test-slave-n2th krb5kdc[14616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394435, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.194@KRBTEST.COM for kudu/127.0.194.254@KRBTEST.COM
I20260926 03:47:16.011878 15390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:15.992544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.194:46501 (local address 127.0.194.254:34561)
0926 03:47:15.992772 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0926 03:47:15.992782 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 03:47:15.994274 (+ 1492us) server_negotiation.cc:407] Connection header received
0926 03:47:15.996435 (+ 2161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:47:15.996444 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:47:15.996570 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:47:15.996704 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:47:15.998438 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:47:15.999228 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:47:16.001105 (+ 1877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:47:16.001835 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:47:16.001982 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:47:16.005708 (+ 3726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:47:16.005713 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:47:16.005716 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:47:16.005752 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:47:16.008634 (+ 2882us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:47:16.009418 (+ 784us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:47:16.009424 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:47:16.009428 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:47:16.009566 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:47:16.010015 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:47:16.010020 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:47:16.010023 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:47:16.010323 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:47:16.010484 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0926 03:47:16.011288 (+ 804us) server_negotiation.cc:299] Negotiation successful
0926 03:47:16.011529 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 03:47:16.013043 15522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:15.993141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:34561 (local address 127.0.194.194:46501)
0926 03:47:15.994058 (+ 917us) negotiation.cc:107] Waiting for socket to connect
0926 03:47:15.994112 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0926 03:47:15.996051 (+ 1939us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:47:15.996920 (+ 869us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:47:15.996929 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:47:15.997569 (+ 640us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:47:15.998236 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:47:15.998255 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:47:15.999394 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:47:15.999400 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:47:16.000917 (+ 1517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:47:16.000936 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:47:16.002065 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:47:16.002069 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:47:16.002288 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:47:16.003097 (+ 809us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:47:16.003121 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:47:16.005455 (+ 2334us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:47:16.008802 (+ 3347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:47:16.008812 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:47:16.008838 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:47:16.009291 (+ 453us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:47:16.009689 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:47:16.009694 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:47:16.009698 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:47:16.009887 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:47:16.010458 (+ 571us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:47:16.010467 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:47:16.010900 (+ 433us) client_negotiation.cc:769] Sending connection context
0926 03:47:16.011182 (+ 282us) client_negotiation.cc:240] Negotiation successful
0926 03:47:16.011768 (+ 586us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":612,"thread_start_us":421,"threads_started":1}
I20260926 03:47:16.014688 15520 heartbeater.cc:347] Connected to a master server at 127.0.194.254:34561
I20260926 03:47:16.015061 15520 heartbeater.cc:464] Registering TS with master...
I20260926 03:47:16.015820 15520 heartbeater.cc:511] Master 127.0.194.254:34561 requested a full tablet report, sending...
I20260926 03:47:16.017310 15202 ts_manager.cc:194] Registered new tserver with Master: ca26e0bdcc7c4021a1ae2b833aa2695f (127.0.194.194:34653)
I20260926 03:47:16.018146 15202 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.194.194@KRBTEST.COM'} at 127.0.194.194:46501
W20260926 03:47:16.355288 15523 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:47:16.355589 15523 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:47:16.355687 15523 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:47:16.363243 15523 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:47:16.363373 15523 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:47:16.363498 15523 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.194.195
I20260926 03:47:16.373536 15523 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.194.212:45229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/krb5kdc/kudu_127.0.194.195.keytab
--principal=kudu/127.0.194.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.194.195:33219
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.194.195
--webserver_port=33033
--tserver_master_addrs=127.0.194.254:34561
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.194.195
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:47:16.374927 15523 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:47:16.376307 15523 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:47:16.389716 15528 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:47:16.389799 15529 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:47:16.389765 15531 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:47:16.390956 15523 server_base.cc:1034] running on GCE node
I20260926 03:47:16.391611 15523 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:47:16.392547 15523 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:47:16.393801 15523 hybrid_clock.cc:648] HybridClock initialized: now 1790394436393723 us; error 62 us; skew 500 ppm
I20260926 03:47:16.394160 15523 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:47:16 dist-test-slave-n2th krb5kdc[14616](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394436, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:47:16.397455 15523 init.cc:377] Logged in from keytab as kudu/127.0.194.195@KRBTEST.COM (short username kudu)
I20260926 03:47:16.398933 15523 webserver.cc:460] Webserver started at http://127.0.194.195:33033/ using document root <none> and password file <none>
I20260926 03:47:16.399670 15523 fs_manager.cc:359] Metadata directory not provided
I20260926 03:47:16.399792 15523 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:47:16.405319 15523 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.005s
I20260926 03:47:16.408128 15538 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:47:16.409529 15523 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 03:47:16.409710 15523 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/minicluster-data/ts-2/wal
uuid: "453de609dea34f9eb8f70055f5398b7e"
format_stamp: "Formatted at 2026-09-26 03:47:13 on dist-test-slave-n2th"
server_key: "5bcb29a21950bef073643bfa6cba164f"
server_key_iv: "da0f90276eff00af416a0a415022a12d"
server_key_version: "encryptionkey@0"
I20260926 03:47:16.410718 15523 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:47:16.434381 15523 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:47:16.439201 15523 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:47:16.439502 15523 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:47:16.440536 15523 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:47:16.441941 15523 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:47:16.442060 15523 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:47:16.442193 15523 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:47:16.442284 15523 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:47:16.478127 15523 rpc_server.cc:307] RPC server started. Bound to: 127.0.194.195:33219
I20260926 03:47:16.478186 15650 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.194.195:33219 every 8 connection(s)
I20260926 03:47:16.479758 15523 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/minicluster-data/ts-2/data/info.pb
I20260926 03:47:16.486311 779 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu as pid 15523
Sep 26 03:47:16 dist-test-slave-n2th krb5kdc[14616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394436, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.195@KRBTEST.COM for kudu/127.0.194.254@KRBTEST.COM
I20260926 03:47:16.503757 15655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:16.487322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59644 (local address 127.0.194.254:34561)
0926 03:47:16.487827 (+ 505us) server_negotiation.cc:206] Beginning negotiation
0926 03:47:16.487832 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:47:16.488367 (+ 535us) server_negotiation.cc:407] Connection header received
0926 03:47:16.488550 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:47:16.488554 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:47:16.488608 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:47:16.488729 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:47:16.490375 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:47:16.491243 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:47:16.492650 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:47:16.493229 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:47:16.493374 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:47:16.496364 (+ 2990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:47:16.496370 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:47:16.496374 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:47:16.496420 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:47:16.500537 (+ 4117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:47:16.501501 (+ 964us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:47:16.501505 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:47:16.501507 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:47:16.501705 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:47:16.502158 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:47:16.502161 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:47:16.502163 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:47:16.502566 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:47:16.502849 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0926 03:47:16.503198 (+ 349us) server_negotiation.cc:299] Negotiation successful
0926 03:47:16.503395 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":99,"server-negotiator.queue_time_us":377,"spinlock_wait_cycles":6400,"thread_start_us":263,"threads_started":1}
I20260926 03:47:16.504305 15390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:16.482521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.195:56901 (local address 127.0.194.254:34561)
0926 03:47:16.482770 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0926 03:47:16.482776 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:47:16.484165 (+ 1389us) server_negotiation.cc:407] Connection header received
0926 03:47:16.486212 (+ 2047us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:47:16.486217 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:47:16.486334 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:47:16.486462 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:47:16.488735 (+ 2273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:47:16.489543 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:47:16.491335 (+ 1792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:47:16.491836 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:47:16.491974 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:47:16.496141 (+ 4167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:47:16.496149 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:47:16.496155 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:47:16.496205 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:47:16.500514 (+ 4309us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:47:16.501617 (+ 1103us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:47:16.501621 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:47:16.501625 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:47:16.501759 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:47:16.502287 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:47:16.502291 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:47:16.502294 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:47:16.502553 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:47:16.502719 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0926 03:47:16.503713 (+ 994us) server_negotiation.cc:299] Negotiation successful
0926 03:47:16.504084 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":82}
I20260926 03:47:16.505609 15653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:16.483222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:34561 (local address 127.0.194.195:56901)
0926 03:47:16.483991 (+ 769us) negotiation.cc:107] Waiting for socket to connect
0926 03:47:16.484037 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 03:47:16.485852 (+ 1815us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:47:16.486795 (+ 943us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:47:16.486836 (+ 41us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:47:16.487614 (+ 778us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:47:16.488551 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:47:16.488579 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:47:16.489712 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:47:16.489718 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:47:16.491176 (+ 1458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:47:16.491190 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:47:16.492097 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:47:16.492104 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:47:16.492358 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:47:16.493091 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:47:16.493119 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:47:16.495860 (+ 2741us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:47:16.500742 (+ 4882us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:47:16.500753 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:47:16.500774 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:47:16.501487 (+ 713us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:47:16.501870 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:47:16.501874 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:47:16.501877 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:47:16.502174 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:47:16.502717 (+ 543us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:47:16.502724 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:47:16.503243 (+ 519us) client_negotiation.cc:769] Sending connection context
0926 03:47:16.503591 (+ 348us) client_negotiation.cc:240] Negotiation successful
0926 03:47:16.504197 (+ 606us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":294,"threads_started":1}
I20260926 03:47:16.507112 15651 heartbeater.cc:347] Connected to a master server at 127.0.194.254:34561
I20260926 03:47:16.507437 15651 heartbeater.cc:464] Registering TS with master...
I20260926 03:47:16.508177 15651 heartbeater.cc:511] Master 127.0.194.254:34561 requested a full tablet report, sending...
I20260926 03:47:16.509716 15201 ts_manager.cc:194] Registered new tserver with Master: 453de609dea34f9eb8f70055f5398b7e (127.0.194.195:33219)
I20260926 03:47:16.510664 15201 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.194.195@KRBTEST.COM'} at 127.0.194.195:56901
I20260926 03:47:16.519377 779 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 03:47:16.527191 15390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:16.520556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59654 (local address 127.0.194.254:34561)
0926 03:47:16.520752 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0926 03:47:16.520761 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:47:16.521156 (+ 395us) server_negotiation.cc:407] Connection header received
0926 03:47:16.521357 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:47:16.521363 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:47:16.521539 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:47:16.521686 (+ 147us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:47:16.522108 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:47:16.523275 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:47:16.524871 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:47:16.525512 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:47:16.525641 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:47:16.526153 (+ 512us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:47:16.526637 (+ 484us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:47:16.526816 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0926 03:47:16.526919 (+ 103us) server_negotiation.cc:299] Negotiation successful
0926 03:47:16.526995 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 03:47:16.528193 15201 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:59654:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 03:47:16.535682 15201 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 03:47:16.561415 15388 heartbeater.cc:503] Master 127.0.194.254:34561 was elected leader, sending a full tablet report...
I20260926 03:47:16.562698 15664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:16.547390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.195:33219 (local address 127.0.0.1:37464)
0926 03:47:16.553522 (+ 6132us) negotiation.cc:107] Waiting for socket to connect
0926 03:47:16.553547 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0926 03:47:16.554454 (+ 907us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:47:16.555535 (+ 1081us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:47:16.555540 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:47:16.555566 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:47:16.556231 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:47:16.556243 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:47:16.557597 (+ 1354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:47:16.557605 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:47:16.560007 (+ 2402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:47:16.560023 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:47:16.561589 (+ 1566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:47:16.561595 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:47:16.561795 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:47:16.561865 (+ 70us) client_negotiation.cc:769] Sending connection context
0926 03:47:16.562016 (+ 151us) client_negotiation.cc:240] Negotiation successful
0926 03:47:16.562232 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5876,"mutex_wait_us":65,"thread_start_us":6513,"threads_started":1}
I20260926 03:47:16.563534 15665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:16.548024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37464 (local address 127.0.194.195:33219)
0926 03:47:16.550772 (+ 2748us) server_negotiation.cc:206] Beginning negotiation
0926 03:47:16.550785 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 03:47:16.553707 (+ 2922us) server_negotiation.cc:407] Connection header received
0926 03:47:16.554687 (+ 980us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:47:16.554694 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:47:16.555022 (+ 328us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:47:16.555214 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:47:16.556429 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:47:16.557461 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:47:16.560208 (+ 2747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:47:16.561390 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:47:16.561540 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:47:16.561802 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0926 03:47:16.562988 (+ 1186us) server_negotiation.cc:299] Negotiation successful
0926 03:47:16.563200 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2489,"thread_start_us":2295,"threads_started":1}
I20260926 03:47:16.563555 15662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:16.547809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.193:38899 (local address 127.0.0.1:50288)
0926 03:47:16.554226 (+ 6417us) negotiation.cc:107] Waiting for socket to connect
0926 03:47:16.554245 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 03:47:16.554389 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:47:16.555488 (+ 1099us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:47:16.555493 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:47:16.555518 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:47:16.556150 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:47:16.556162 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:47:16.558284 (+ 2122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:47:16.558288 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:47:16.560305 (+ 2017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:47:16.560321 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:47:16.562665 (+ 2344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:47:16.562673 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:47:16.562910 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:47:16.562940 (+ 30us) client_negotiation.cc:769] Sending connection context
0926 03:47:16.563025 (+ 85us) client_negotiation.cc:240] Negotiation successful
0926 03:47:16.563161 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6276,"mutex_wait_us":90,"thread_start_us":5299,"threads_started":1}
I20260926 03:47:16.564100 15666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:16.548471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50288 (local address 127.0.194.193:38899)
0926 03:47:16.552863 (+ 4392us) server_negotiation.cc:206] Beginning negotiation
0926 03:47:16.552875 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 03:47:16.554355 (+ 1480us) server_negotiation.cc:407] Connection header received
0926 03:47:16.554666 (+ 311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:47:16.554672 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:47:16.554962 (+ 290us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:47:16.555196 (+ 234us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:47:16.556352 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:47:16.558081 (+ 1729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:47:16.560468 (+ 2387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:47:16.562041 (+ 1573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:47:16.562176 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:47:16.562410 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0926 03:47:16.563601 (+ 1191us) server_negotiation.cc:299] Negotiation successful
0926 03:47:16.563791 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4165,"thread_start_us":3946,"threads_started":1}
I20260926 03:47:16.564989 15663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:16.547572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40074 (local address 127.0.194.194:34653)
0926 03:47:16.554491 (+ 6919us) server_negotiation.cc:206] Beginning negotiation
0926 03:47:16.554499 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:47:16.554526 (+ 27us) server_negotiation.cc:407] Connection header received
0926 03:47:16.554646 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:47:16.554655 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:47:16.554963 (+ 308us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:47:16.556199 (+ 1236us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:47:16.556434 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:47:16.558096 (+ 1662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:47:16.560754 (+ 2658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:47:16.563496 (+ 2742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:47:16.564056 (+ 560us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:47:16.564419 (+ 363us) server_negotiation.cc:1035] Waiting for connection context
0926 03:47:16.564497 (+ 78us) server_negotiation.cc:299] Negotiation successful
0926 03:47:16.564667 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6735,"thread_start_us":6593,"threads_started":1}
I20260926 03:47:16.565021 15661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:16.547195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.194:34653 (local address 127.0.0.1:40074)
0926 03:47:16.549608 (+ 2413us) negotiation.cc:107] Waiting for socket to connect
0926 03:47:16.549669 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0926 03:47:16.549893 (+ 224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:47:16.555272 (+ 5379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:47:16.555282 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:47:16.555312 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:47:16.555935 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:47:16.555951 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:47:16.558238 (+ 2287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:47:16.558243 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:47:16.560582 (+ 2339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:47:16.560595 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:47:16.564145 (+ 3550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:47:16.564154 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:47:16.564336 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:47:16.564370 (+ 34us) client_negotiation.cc:769] Sending connection context
0926 03:47:16.564455 (+ 85us) client_negotiation.cc:240] Negotiation successful
0926 03:47:16.564592 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2064,"mutex_wait_us":30,"thread_start_us":2250,"threads_started":1}
I20260926 03:47:16.567766 15586 tablet_service.cc:1467] Processing CreateTablet for tablet 6bfdc439a5194d7eb3a2cd848d9fbd39 (DEFAULT_TABLE table=test-table [id=771a89ce3ebd42e29af2cfc9590dffc3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:47:16.568320 15455 tablet_service.cc:1467] Processing CreateTablet for tablet 6bfdc439a5194d7eb3a2cd848d9fbd39 (DEFAULT_TABLE table=test-table [id=771a89ce3ebd42e29af2cfc9590dffc3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:47:16.569320 15586 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6bfdc439a5194d7eb3a2cd848d9fbd39. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:47:16.568903 15323 tablet_service.cc:1467] Processing CreateTablet for tablet 6bfdc439a5194d7eb3a2cd848d9fbd39 (DEFAULT_TABLE table=test-table [id=771a89ce3ebd42e29af2cfc9590dffc3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:47:16.569533 15455 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6bfdc439a5194d7eb3a2cd848d9fbd39. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:47:16.570211 15323 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6bfdc439a5194d7eb3a2cd848d9fbd39. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:47:16.583498 15667 tablet_bootstrap.cc:492] T 6bfdc439a5194d7eb3a2cd848d9fbd39 P ca26e0bdcc7c4021a1ae2b833aa2695f: Bootstrap starting.
I20260926 03:47:16.585798 15668 tablet_bootstrap.cc:492] T 6bfdc439a5194d7eb3a2cd848d9fbd39 P 26e7194486b140e2b7434d73bbe399e6: Bootstrap starting.
I20260926 03:47:16.587649 15669 tablet_bootstrap.cc:492] T 6bfdc439a5194d7eb3a2cd848d9fbd39 P 453de609dea34f9eb8f70055f5398b7e: Bootstrap starting.
I20260926 03:47:16.589562 15667 tablet_bootstrap.cc:654] T 6bfdc439a5194d7eb3a2cd848d9fbd39 P ca26e0bdcc7c4021a1ae2b833aa2695f: Neither blocks nor log segments found. Creating new log.
I20260926 03:47:16.589567 15668 tablet_bootstrap.cc:654] T 6bfdc439a5194d7eb3a2cd848d9fbd39 P 26e7194486b140e2b7434d73bbe399e6: Neither blocks nor log segments found. Creating new log.
I20260926 03:47:16.590552 15669 tablet_bootstrap.cc:654] T 6bfdc439a5194d7eb3a2cd848d9fbd39 P 453de609dea34f9eb8f70055f5398b7e: Neither blocks nor log segments found. Creating new log.
I20260926 03:47:16.590955 15668 log.cc:824] T 6bfdc439a5194d7eb3a2cd848d9fbd39 P 26e7194486b140e2b7434d73bbe399e6: Log is configured to *not* fsync() on all Append() calls
I20260926 03:47:16.590955 15667 log.cc:824] T 6bfdc439a5194d7eb3a2cd848d9fbd39 P ca26e0bdcc7c4021a1ae2b833aa2695f: Log is configured to *not* fsync() on all Append() calls
I20260926 03:47:16.591639 15669 log.cc:824] T 6bfdc439a5194d7eb3a2cd848d9fbd39 P 453de609dea34f9eb8f70055f5398b7e: Log is configured to *not* fsync() on all Append() calls
I20260926 03:47:16.594527 15667 tablet_bootstrap.cc:492] T 6bfdc439a5194d7eb3a2cd848d9fbd39 P ca26e0bdcc7c4021a1ae2b833aa2695f: No bootstrap required, opened a new log
I20260926 03:47:16.594527 15668 tablet_bootstrap.cc:492] T 6bfdc439a5194d7eb3a2cd848d9fbd39 P 26e7194486b140e2b7434d73bbe399e6: No bootstrap required, opened a new log
I20260926 03:47:16.594847 15668 ts_tablet_manager.cc:1397] T 6bfdc439a5194d7eb3a2cd848d9fbd39 P 26e7194486b140e2b7434d73bbe399e6: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260926 03:47:16.594847 15667 ts_tablet_manager.cc:1397] T 6bfdc439a5194d7eb3a2cd848d9fbd39 P ca26e0bdcc7c4021a1ae2b833aa2695f: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.004s
I20260926 03:47:16.594859 15669 tablet_bootstrap.cc:492] T 6bfdc439a5194d7eb3a2cd848d9fbd39 P 453de609dea34f9eb8f70055f5398b7e: No bootstrap required, opened a new log
I20260926 03:47:16.595141 15669 ts_tablet_manager.cc:1397] T 6bfdc439a5194d7eb3a2cd848d9fbd39 P 453de609dea34f9eb8f70055f5398b7e: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260926 03:47:16.602875 15669 raft_consensus.cc:348] T 6bfdc439a5194d7eb3a2cd848d9fbd39 P 453de609dea34f9eb8f70055f5398b7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca26e0bdcc7c4021a1ae2b833aa2695f" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 34653 } } peers { permanent_uuid: "453de609dea34f9eb8f70055f5398b7e" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 33219 } } peers { permanent_uuid: "26e7194486b140e2b7434d73bbe399e6" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 38899 } }
I20260926 03:47:16.603302 15669 raft_consensus.cc:374] T 6bfdc439a5194d7eb3a2cd848d9fbd39 P 453de609dea34f9eb8f70055f5398b7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:47:16.603406 15669 raft_consensus.cc:729] T 6bfdc439a5194d7eb3a2cd848d9fbd39 P 453de609dea34f9eb8f70055f5398b7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 453de609dea34f9eb8f70055f5398b7e, State: Initialized, Role: FOLLOWER
I20260926 03:47:16.604063 15669 consensus_queue.cc:260] T 6bfdc439a5194d7eb3a2cd848d9fbd39 P 453de609dea34f9eb8f70055f5398b7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca26e0bdcc7c4021a1ae2b833aa2695f" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 34653 } } peers { permanent_uuid: "453de609dea34f9eb8f70055f5398b7e" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 33219 } } peers { permanent_uuid: "26e7194486b140e2b7434d73bbe399e6" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 38899 } }
I20260926 03:47:16.605291 15651 heartbeater.cc:503] Master 127.0.194.254:34561 was elected leader, sending a full tablet report...
I20260926 03:47:16.605841 15669 ts_tablet_manager.cc:1428] T 6bfdc439a5194d7eb3a2cd848d9fbd39 P 453de609dea34f9eb8f70055f5398b7e: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
I20260926 03:47:16.606055 15668 raft_consensus.cc:348] T 6bfdc439a5194d7eb3a2cd848d9fbd39 P 26e7194486b140e2b7434d73bbe399e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca26e0bdcc7c4021a1ae2b833aa2695f" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 34653 } } peers { permanent_uuid: "453de609dea34f9eb8f70055f5398b7e" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 33219 } } peers { permanent_uuid: "26e7194486b140e2b7434d73bbe399e6" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 38899 } }
I20260926 03:47:16.606515 15668 raft_consensus.cc:374] T 6bfdc439a5194d7eb3a2cd848d9fbd39 P 26e7194486b140e2b7434d73bbe399e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:47:16.606644 15668 raft_consensus.cc:729] T 6bfdc439a5194d7eb3a2cd848d9fbd39 P 26e7194486b140e2b7434d73bbe399e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26e7194486b140e2b7434d73bbe399e6, State: Initialized, Role: FOLLOWER
I20260926 03:47:16.607105 15667 raft_consensus.cc:348] T 6bfdc439a5194d7eb3a2cd848d9fbd39 P ca26e0bdcc7c4021a1ae2b833aa2695f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca26e0bdcc7c4021a1ae2b833aa2695f" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 34653 } } peers { permanent_uuid: "453de609dea34f9eb8f70055f5398b7e" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 33219 } } peers { permanent_uuid: "26e7194486b140e2b7434d73bbe399e6" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 38899 } }
I20260926 03:47:16.607311 15668 consensus_queue.cc:260] T 6bfdc439a5194d7eb3a2cd848d9fbd39 P 26e7194486b140e2b7434d73bbe399e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca26e0bdcc7c4021a1ae2b833aa2695f" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 34653 } } peers { permanent_uuid: "453de609dea34f9eb8f70055f5398b7e" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 33219 } } peers { permanent_uuid: "26e7194486b140e2b7434d73bbe399e6" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 38899 } }
I20260926 03:47:16.607595 15667 raft_consensus.cc:374] T 6bfdc439a5194d7eb3a2cd848d9fbd39 P ca26e0bdcc7c4021a1ae2b833aa2695f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:47:16.607698 15667 raft_consensus.cc:729] T 6bfdc439a5194d7eb3a2cd848d9fbd39 P ca26e0bdcc7c4021a1ae2b833aa2695f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca26e0bdcc7c4021a1ae2b833aa2695f, State: Initialized, Role: FOLLOWER
I20260926 03:47:16.609172 15668 ts_tablet_manager.cc:1428] T 6bfdc439a5194d7eb3a2cd848d9fbd39 P 26e7194486b140e2b7434d73bbe399e6: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260926 03:47:16.609608 15667 consensus_queue.cc:260] T 6bfdc439a5194d7eb3a2cd848d9fbd39 P ca26e0bdcc7c4021a1ae2b833aa2695f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca26e0bdcc7c4021a1ae2b833aa2695f" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 34653 } } peers { permanent_uuid: "453de609dea34f9eb8f70055f5398b7e" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 33219 } } peers { permanent_uuid: "26e7194486b140e2b7434d73bbe399e6" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 38899 } }
I20260926 03:47:16.610834 15520 heartbeater.cc:503] Master 127.0.194.254:34561 was elected leader, sending a full tablet report...
I20260926 03:47:16.611521 15667 ts_tablet_manager.cc:1428] T 6bfdc439a5194d7eb3a2cd848d9fbd39 P ca26e0bdcc7c4021a1ae2b833aa2695f: Time spent starting tablet: real 0.017s user 0.014s sys 0.000s
I20260926 03:47:16.675052 15674 raft_consensus.cc:482] T 6bfdc439a5194d7eb3a2cd848d9fbd39 P 26e7194486b140e2b7434d73bbe399e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 03:47:16.675372 15674 raft_consensus.cc:504] T 6bfdc439a5194d7eb3a2cd848d9fbd39 P 26e7194486b140e2b7434d73bbe399e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca26e0bdcc7c4021a1ae2b833aa2695f" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 34653 } } peers { permanent_uuid: "453de609dea34f9eb8f70055f5398b7e" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 33219 } } peers { permanent_uuid: "26e7194486b140e2b7434d73bbe399e6" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 38899 } }
I20260926 03:47:16.677054 15674 leader_election.cc:288] T 6bfdc439a5194d7eb3a2cd848d9fbd39 P 26e7194486b140e2b7434d73bbe399e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ca26e0bdcc7c4021a1ae2b833aa2695f (127.0.194.194:34653), 453de609dea34f9eb8f70055f5398b7e (127.0.194.195:33219)
I20260926 03:47:16.686518 15665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:16.677508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.193:37883 (local address 127.0.194.195:33219)
0926 03:47:16.677703 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0926 03:47:16.677709 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:47:16.678490 (+ 781us) server_negotiation.cc:407] Connection header received
0926 03:47:16.678697 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:47:16.678703 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:47:16.678822 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:47:16.678916 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:47:16.679690 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:47:16.680840 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:47:16.683968 (+ 3128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:47:16.685350 (+ 1382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:47:16.685512 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:47:16.685584 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 03:47:16.686118 (+ 534us) server_negotiation.cc:299] Negotiation successful
0926 03:47:16.686253 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 03:47:16.686518 15663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:16.677501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.193:60797 (local address 127.0.194.194:34653)
0926 03:47:16.677703 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0926 03:47:16.677709 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:47:16.678489 (+ 780us) server_negotiation.cc:407] Connection header received
0926 03:47:16.678717 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:47:16.678722 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:47:16.678821 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:47:16.678922 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:47:16.679752 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:47:16.680840 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:47:16.683968 (+ 3128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:47:16.685330 (+ 1362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:47:16.685508 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:47:16.685584 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 03:47:16.686118 (+ 534us) server_negotiation.cc:299] Negotiation successful
0926 03:47:16.686253 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 03:47:16.686790 15676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:16.677433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.194:34653 (local address 127.0.194.193:60797)
0926 03:47:16.678352 (+ 919us) negotiation.cc:107] Waiting for socket to connect
0926 03:47:16.678375 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 03:47:16.678592 (+ 217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:47:16.678953 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:47:16.678956 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:47:16.679001 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:47:16.679590 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:47:16.679603 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:47:16.680982 (+ 1379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:47:16.680987 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:47:16.683788 (+ 2801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:47:16.683799 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:47:16.685594 (+ 1795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:47:16.685606 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:47:16.685867 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:47:16.685952 (+ 85us) client_negotiation.cc:769] Sending connection context
0926 03:47:16.686055 (+ 103us) client_negotiation.cc:240] Negotiation successful
0926 03:47:16.686226 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":801,"mutex_wait_us":166,"spinlock_wait_cycles":46080,"thread_start_us":506,"threads_started":1}
I20260926 03:47:16.687186 15677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:16.677335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.195:33219 (local address 127.0.194.193:37883)
0926 03:47:16.678352 (+ 1017us) negotiation.cc:107] Waiting for socket to connect
0926 03:47:16.678374 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 03:47:16.678538 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:47:16.678948 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:47:16.678954 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:47:16.679016 (+ 62us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:47:16.679522 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:47:16.679541 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:47:16.680983 (+ 1442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:47:16.680987 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:47:16.683781 (+ 2794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:47:16.683800 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:47:16.685624 (+ 1824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:47:16.685630 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:47:16.685875 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:47:16.685953 (+ 78us) client_negotiation.cc:769] Sending connection context
0926 03:47:16.686055 (+ 102us) client_negotiation.cc:240] Negotiation successful
0926 03:47:16.686270 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":831,"mutex_wait_us":330,"spinlock_wait_cycles":53120,"thread_start_us":607,"threads_started":1}
I20260926 03:47:16.687777 15475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bfdc439a5194d7eb3a2cd848d9fbd39" candidate_uuid: "26e7194486b140e2b7434d73bbe399e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca26e0bdcc7c4021a1ae2b833aa2695f" is_pre_election: true
I20260926 03:47:16.688109 15606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bfdc439a5194d7eb3a2cd848d9fbd39" candidate_uuid: "26e7194486b140e2b7434d73bbe399e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "453de609dea34f9eb8f70055f5398b7e" is_pre_election: true
I20260926 03:47:16.688271 15475 raft_consensus.cc:2454] T 6bfdc439a5194d7eb3a2cd848d9fbd39 P ca26e0bdcc7c4021a1ae2b833aa2695f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26e7194486b140e2b7434d73bbe399e6 in term 0.
I20260926 03:47:16.688607 15606 raft_consensus.cc:2454] T 6bfdc439a5194d7eb3a2cd848d9fbd39 P 453de609dea34f9eb8f70055f5398b7e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26e7194486b140e2b7434d73bbe399e6 in term 0.
I20260926 03:47:16.688959 15276 leader_election.cc:302] T 6bfdc439a5194d7eb3a2cd848d9fbd39 P 26e7194486b140e2b7434d73bbe399e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26e7194486b140e2b7434d73bbe399e6, ca26e0bdcc7c4021a1ae2b833aa2695f; no voters:
I20260926 03:47:16.689353 15674 raft_consensus.cc:2781] T 6bfdc439a5194d7eb3a2cd848d9fbd39 P 26e7194486b140e2b7434d73bbe399e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 03:47:16.689497 15674 raft_consensus.cc:482] T 6bfdc439a5194d7eb3a2cd848d9fbd39 P 26e7194486b140e2b7434d73bbe399e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 03:47:16.689611 15674 raft_consensus.cc:3037] T 6bfdc439a5194d7eb3a2cd848d9fbd39 P 26e7194486b140e2b7434d73bbe399e6 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:47:16.691912 15674 raft_consensus.cc:504] T 6bfdc439a5194d7eb3a2cd848d9fbd39 P 26e7194486b140e2b7434d73bbe399e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca26e0bdcc7c4021a1ae2b833aa2695f" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 34653 } } peers { permanent_uuid: "453de609dea34f9eb8f70055f5398b7e" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 33219 } } peers { permanent_uuid: "26e7194486b140e2b7434d73bbe399e6" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 38899 } }
I20260926 03:47:16.692588 15674 leader_election.cc:288] T 6bfdc439a5194d7eb3a2cd848d9fbd39 P 26e7194486b140e2b7434d73bbe399e6 [CANDIDATE]: Term 1 election: Requested vote from peers ca26e0bdcc7c4021a1ae2b833aa2695f (127.0.194.194:34653), 453de609dea34f9eb8f70055f5398b7e (127.0.194.195:33219)
I20260926 03:47:16.693017 15475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bfdc439a5194d7eb3a2cd848d9fbd39" candidate_uuid: "26e7194486b140e2b7434d73bbe399e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca26e0bdcc7c4021a1ae2b833aa2695f"
I20260926 03:47:16.693123 15606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bfdc439a5194d7eb3a2cd848d9fbd39" candidate_uuid: "26e7194486b140e2b7434d73bbe399e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "453de609dea34f9eb8f70055f5398b7e"
I20260926 03:47:16.693221 15475 raft_consensus.cc:3037] T 6bfdc439a5194d7eb3a2cd848d9fbd39 P ca26e0bdcc7c4021a1ae2b833aa2695f [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:47:16.693374 15606 raft_consensus.cc:3037] T 6bfdc439a5194d7eb3a2cd848d9fbd39 P 453de609dea34f9eb8f70055f5398b7e [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:47:16.695843 15475 raft_consensus.cc:2454] T 6bfdc439a5194d7eb3a2cd848d9fbd39 P ca26e0bdcc7c4021a1ae2b833aa2695f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26e7194486b140e2b7434d73bbe399e6 in term 1.
I20260926 03:47:16.695854 15606 raft_consensus.cc:2454] T 6bfdc439a5194d7eb3a2cd848d9fbd39 P 453de609dea34f9eb8f70055f5398b7e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26e7194486b140e2b7434d73bbe399e6 in term 1.
I20260926 03:47:16.696336 15279 leader_election.cc:302] T 6bfdc439a5194d7eb3a2cd848d9fbd39 P 26e7194486b140e2b7434d73bbe399e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26e7194486b140e2b7434d73bbe399e6, 453de609dea34f9eb8f70055f5398b7e; no voters:
I20260926 03:47:16.696691 15674 raft_consensus.cc:2781] T 6bfdc439a5194d7eb3a2cd848d9fbd39 P 26e7194486b140e2b7434d73bbe399e6 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:47:16.697150 15674 raft_consensus.cc:686] T 6bfdc439a5194d7eb3a2cd848d9fbd39 P 26e7194486b140e2b7434d73bbe399e6 [term 1 LEADER]: Becoming Leader. State: Replica: 26e7194486b140e2b7434d73bbe399e6, State: Running, Role: LEADER
I20260926 03:47:16.697633 15674 consensus_queue.cc:237] T 6bfdc439a5194d7eb3a2cd848d9fbd39 P 26e7194486b140e2b7434d73bbe399e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca26e0bdcc7c4021a1ae2b833aa2695f" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 34653 } } peers { permanent_uuid: "453de609dea34f9eb8f70055f5398b7e" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 33219 } } peers { permanent_uuid: "26e7194486b140e2b7434d73bbe399e6" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 38899 } }
I20260926 03:47:16.702888 15201 catalog_manager.cc:5495] T 6bfdc439a5194d7eb3a2cd848d9fbd39 P 26e7194486b140e2b7434d73bbe399e6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 26e7194486b140e2b7434d73bbe399e6 (127.0.194.193). New cstate: current_term: 1 leader_uuid: "26e7194486b140e2b7434d73bbe399e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca26e0bdcc7c4021a1ae2b833aa2695f" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 34653 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "453de609dea34f9eb8f70055f5398b7e" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 33219 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26e7194486b140e2b7434d73bbe399e6" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 38899 } health_report { overall_health: HEALTHY } } }
W20260926 03:47:16.731429 15652 tablet.cc:2369] T 6bfdc439a5194d7eb3a2cd848d9fbd39 P 453de609dea34f9eb8f70055f5398b7e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 03:47:16.741839 15521 tablet.cc:2369] T 6bfdc439a5194d7eb3a2cd848d9fbd39 P ca26e0bdcc7c4021a1ae2b833aa2695f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 03:47:16.751456 15666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:16.746294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50302 (local address 127.0.194.193:38899)
0926 03:47:16.746575 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0926 03:47:16.746580 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:47:16.746599 (+ 19us) server_negotiation.cc:407] Connection header received
0926 03:47:16.746767 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:47:16.746770 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:47:16.746861 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:47:16.746975 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:47:16.747313 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:47:16.748019 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:47:16.749383 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:47:16.749784 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:47:16.749877 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:47:16.750354 (+ 477us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:47:16.750880 (+ 526us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:47:16.750984 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0926 03:47:16.751074 (+ 90us) server_negotiation.cc:299] Negotiation successful
0926 03:47:16.751141 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122}
I20260926 03:47:16.760090 15606 raft_consensus.cc:1261] T 6bfdc439a5194d7eb3a2cd848d9fbd39 P 453de609dea34f9eb8f70055f5398b7e [term 1 FOLLOWER]: Refusing update from remote peer 26e7194486b140e2b7434d73bbe399e6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:47:16.760090 15475 raft_consensus.cc:1261] T 6bfdc439a5194d7eb3a2cd848d9fbd39 P ca26e0bdcc7c4021a1ae2b833aa2695f [term 1 FOLLOWER]: Refusing update from remote peer 26e7194486b140e2b7434d73bbe399e6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:47:16.761199 15678 consensus_queue.cc:1035] T 6bfdc439a5194d7eb3a2cd848d9fbd39 P 26e7194486b140e2b7434d73bbe399e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca26e0bdcc7c4021a1ae2b833aa2695f" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 34653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:47:16.761545 15674 consensus_queue.cc:1035] T 6bfdc439a5194d7eb3a2cd848d9fbd39 P 26e7194486b140e2b7434d73bbe399e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453de609dea34f9eb8f70055f5398b7e" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 33219 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:47:16.774529 15682 mvcc.cc:205] Tried to move back new op lower bound from 7333455612959105024 to 7333455612717522944. Current Snapshot: MvccSnapshot[applied={T|T < 7333455612959105024}]
I20260926 03:47:16.780921 15681 mvcc.cc:205] Tried to move back new op lower bound from 7333455612959105024 to 7333455612717522944. Current Snapshot: MvccSnapshot[applied={T|T < 7333455612959105024}]
I20260926 03:47:16.788234 15202 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:59654:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:47:16.788486 15202 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:59654:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:47:16.792223 15202 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fd19af12fdda4aa9be763b660e4611c8: Sending DeleteTablet for 3 replicas of tablet 6bfdc439a5194d7eb3a2cd848d9fbd39
I20260926 03:47:16.793560 15455 tablet_service.cc:1514] Processing DeleteTablet for tablet 6bfdc439a5194d7eb3a2cd848d9fbd39 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:47:16 UTC) from {username='kudu', principal='kudu/127.0.194.254@KRBTEST.COM'} at 127.0.0.1:40074
I20260926 03:47:16.793694 15323 tablet_service.cc:1514] Processing DeleteTablet for tablet 6bfdc439a5194d7eb3a2cd848d9fbd39 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:47:16 UTC) from {username='kudu', principal='kudu/127.0.194.254@KRBTEST.COM'} at 127.0.0.1:50288
I20260926 03:47:16.793713 15586 tablet_service.cc:1514] Processing DeleteTablet for tablet 6bfdc439a5194d7eb3a2cd848d9fbd39 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:47:16 UTC) from {username='kudu', principal='kudu/127.0.194.254@KRBTEST.COM'} at 127.0.0.1:37464
I20260926 03:47:16.797025 779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu pid 15239
I20260926 03:47:16.821643 15666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:16.798660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50304 (local address 127.0.194.193:38899)
0926 03:47:16.798858 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0926 03:47:16.798864 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:47:16.798882 (+ 18us) server_negotiation.cc:407] Connection header received
0926 03:47:16.798943 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:47:16.798949 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:47:16.799049 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:47:16.799165 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:47:16.801327 (+ 2162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:47:16.802562 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:47:16.804890 (+ 2328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:47:16.805665 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:47:16.805855 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:47:16.811105 (+ 5250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:47:16.811121 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:47:16.811143 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:47:16.811216 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:47:16.815703 (+ 4487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:47:16.816799 (+ 1096us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:47:16.816811 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:47:16.816822 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:47:16.817017 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:47:16.818384 (+ 1367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:47:16.818389 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:47:16.818392 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:47:16.818933 (+ 541us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:47:16.819251 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0926 03:47:16.820961 (+ 1710us) server_negotiation.cc:299] Negotiation successful
0926 03:47:16.821224 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260926 03:47:16.822392 15383 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:47:16.828074 15692 tablet_replica.cc:330] T 6bfdc439a5194d7eb3a2cd848d9fbd39 P 453de609dea34f9eb8f70055f5398b7e: stopping tablet replica
I20260926 03:47:16.828696 15692 raft_consensus.cc:2229] T 6bfdc439a5194d7eb3a2cd848d9fbd39 P 453de609dea34f9eb8f70055f5398b7e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:47:16.829352 15692 raft_consensus.cc:2258] T 6bfdc439a5194d7eb3a2cd848d9fbd39 P 453de609dea34f9eb8f70055f5398b7e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:47:16.830411 15691 tablet_replica.cc:330] T 6bfdc439a5194d7eb3a2cd848d9fbd39 P ca26e0bdcc7c4021a1ae2b833aa2695f: stopping tablet replica
I20260926 03:47:16.830854 15691 raft_consensus.cc:2229] T 6bfdc439a5194d7eb3a2cd848d9fbd39 P ca26e0bdcc7c4021a1ae2b833aa2695f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:47:16.831334 15691 raft_consensus.cc:2258] T 6bfdc439a5194d7eb3a2cd848d9fbd39 P ca26e0bdcc7c4021a1ae2b833aa2695f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:47:16.832803 15692 ts_tablet_manager.cc:1905] T 6bfdc439a5194d7eb3a2cd848d9fbd39 P 453de609dea34f9eb8f70055f5398b7e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:47:16.833137 15691 ts_tablet_manager.cc:1905] T 6bfdc439a5194d7eb3a2cd848d9fbd39 P ca26e0bdcc7c4021a1ae2b833aa2695f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:47:16.840113 15692 ts_tablet_manager.cc:1918] T 6bfdc439a5194d7eb3a2cd848d9fbd39 P 453de609dea34f9eb8f70055f5398b7e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:47:16.840113 15691 ts_tablet_manager.cc:1918] T 6bfdc439a5194d7eb3a2cd848d9fbd39 P ca26e0bdcc7c4021a1ae2b833aa2695f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:47:16.840306 15692 log.cc:1196] T 6bfdc439a5194d7eb3a2cd848d9fbd39 P 453de609dea34f9eb8f70055f5398b7e: Deleting WAL directory at /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/minicluster-data/ts-2/wal/wals/6bfdc439a5194d7eb3a2cd848d9fbd39
I20260926 03:47:16.840306 15691 log.cc:1196] T 6bfdc439a5194d7eb3a2cd848d9fbd39 P ca26e0bdcc7c4021a1ae2b833aa2695f: Deleting WAL directory at /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/minicluster-data/ts-1/wal/wals/6bfdc439a5194d7eb3a2cd848d9fbd39
I20260926 03:47:16.840879 15692 ts_tablet_manager.cc:1939] T 6bfdc439a5194d7eb3a2cd848d9fbd39 P 453de609dea34f9eb8f70055f5398b7e: Deleting consensus metadata
I20260926 03:47:16.840879 15691 ts_tablet_manager.cc:1939] T 6bfdc439a5194d7eb3a2cd848d9fbd39 P ca26e0bdcc7c4021a1ae2b833aa2695f: Deleting consensus metadata
I20260926 03:47:16.842857 15189 catalog_manager.cc:4841] TS 453de609dea34f9eb8f70055f5398b7e (127.0.194.195:33219): tablet 6bfdc439a5194d7eb3a2cd848d9fbd39 (table test-table [id=771a89ce3ebd42e29af2cfc9590dffc3]) successfully deleted
I20260926 03:47:16.842967 15186 catalog_manager.cc:4841] TS ca26e0bdcc7c4021a1ae2b833aa2695f (127.0.194.194:34653): tablet 6bfdc439a5194d7eb3a2cd848d9fbd39 (table test-table [id=771a89ce3ebd42e29af2cfc9590dffc3]) successfully deleted
I20260926 03:47:16.934657 15690 tablet_replica.cc:330] T 6bfdc439a5194d7eb3a2cd848d9fbd39 P 26e7194486b140e2b7434d73bbe399e6: stopping tablet replica
I20260926 03:47:16.934723 779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu with pid 15239
W20260926 03:47:16.953790 15188 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.194.193:38899 (error 108)
I20260926 03:47:16.953936 779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu pid 15392
I20260926 03:47:16.955766 15662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:16.955297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.193:38899 (local address 127.0.0.1:50306)
0926 03:47:16.955480 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0926 03:47:16.955601 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.193:38899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260926 03:47:16.956033 15188 catalog_manager.cc:4584] TS 26e7194486b140e2b7434d73bbe399e6 (127.0.194.193:38899): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6bfdc439a5194d7eb3a2cd848d9fbd39: Network error: Client connection negotiation failed: client connection to 127.0.194.193:38899: connect: Connection refused (error 111)
I20260926 03:47:16.968004 15663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:16.954910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40076 (local address 127.0.194.194:34653)
0926 03:47:16.955106 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0926 03:47:16.955112 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:47:16.955135 (+ 23us) server_negotiation.cc:407] Connection header received
0926 03:47:16.955258 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:47:16.955263 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:47:16.955357 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:47:16.955473 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:47:16.956938 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:47:16.957665 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:47:16.959167 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:47:16.959550 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:47:16.959644 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:47:16.962185 (+ 2541us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:47:16.962197 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:47:16.962213 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:47:16.962244 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:47:16.965139 (+ 2895us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:47:16.965919 (+ 780us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:47:16.965927 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:47:16.965929 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:47:16.966059 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:47:16.966393 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:47:16.966396 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:47:16.966398 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:47:16.966932 (+ 534us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:47:16.967165 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0926 03:47:16.967459 (+ 294us) server_negotiation.cc:299] Negotiation successful
0926 03:47:16.967623 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 03:47:16.968641 15515 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:47:17.009485 15662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:17.009013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.193:38899 (local address 127.0.0.1:50308)
0926 03:47:17.009216 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0926 03:47:17.009334 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.193:38899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260926 03:47:17.010545 15662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:17.010181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.193:38899 (local address 127.0.0.1:50314)
0926 03:47:17.010336 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0926 03:47:17.010430 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.193:38899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260926 03:47:17.071660 15662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:17.071072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.193:38899 (local address 127.0.0.1:50330)
0926 03:47:17.071341 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0926 03:47:17.071470 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.193:38899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260926 03:47:17.072815 15662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:17.072432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.193:38899 (local address 127.0.0.1:50346)
0926 03:47:17.072585 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0926 03:47:17.072676 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.193:38899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260926 03:47:17.075618 779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu with pid 15392
I20260926 03:47:17.095552 779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu pid 15523
I20260926 03:47:17.110867 15665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:17.096455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37474 (local address 127.0.194.195:33219)
0926 03:47:17.096643 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0926 03:47:17.096648 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:47:17.096668 (+ 20us) server_negotiation.cc:407] Connection header received
0926 03:47:17.096905 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:47:17.096910 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:47:17.096999 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:47:17.097095 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:47:17.098567 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:47:17.099407 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:47:17.100731 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:47:17.101181 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:47:17.101284 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:47:17.104540 (+ 3256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:47:17.104556 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:47:17.104573 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:47:17.104619 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:47:17.107815 (+ 3196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:47:17.108640 (+ 825us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:47:17.108649 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:47:17.108651 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:47:17.108814 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:47:17.109257 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:47:17.109261 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:47:17.109263 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:47:17.109667 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:47:17.109912 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0926 03:47:17.110235 (+ 323us) server_negotiation.cc:299] Negotiation successful
0926 03:47:17.110416 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 03:47:17.111548 15646 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:47:17.153339 15662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:17.152815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.193:38899 (local address 127.0.0.1:50358)
0926 03:47:17.153043 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0926 03:47:17.153146 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.193:38899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260926 03:47:17.154469 15662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:17.154056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.193:38899 (local address 127.0.0.1:50374)
0926 03:47:17.154288 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0926 03:47:17.154362 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.193:38899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260926 03:47:17.203282 779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu with pid 15523
I20260926 03:47:17.220376 779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu pid 15169
I20260926 03:47:17.221657 15232 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:47:17.335886 15662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:17.335111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.193:38899 (local address 127.0.0.1:50388)
0926 03:47:17.335336 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0926 03:47:17.335457 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.193:38899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260926 03:47:17.336133 779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu with pid 15169
I20260926 03:47:17.358139 779 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
/tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.194.254:34561
--webserver_interface=127.0.194.254
--webserver_port=37679
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/krb5kdc/kudu_127.0.194.254.keytab
--principal=kudu/127.0.194.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.194.212:45229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.194.254:34561
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:47:17.699383 15698 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:47:17.699743 15698 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:47:17.699913 15698 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:47:17.707613 15698 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:47:17.707789 15698 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:47:17.707901 15698 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:47:17.708001 15698 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:47:17.708086 15698 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:47:17.721827 15698 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.194.212:45229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.194.254:34561
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/krb5kdc/kudu_127.0.194.254.keytab
--principal=kudu/127.0.194.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.194.254:34561
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.194.254
--webserver_port=37679
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:47:17.723376 15698 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:47:17.724785 15698 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:47:17.737195 15703 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:47:17.737238 15706 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:47:17.737157 15704 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:47:17.738509 15698 server_base.cc:1034] running on GCE node
I20260926 03:47:17.739400 15698 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:47:17.740993 15698 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:47:17.742280 15698 hybrid_clock.cc:648] HybridClock initialized: now 1790394437742201 us; error 63 us; skew 500 ppm
I20260926 03:47:17.742782 15698 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:47:17 dist-test-slave-n2th krb5kdc[14616](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394437, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:47:17.746556 15698 init.cc:377] Logged in from keytab as kudu/127.0.194.254@KRBTEST.COM (short username kudu)
I20260926 03:47:17.748299 15698 webserver.cc:460] Webserver started at http://127.0.194.254:37679/ using document root <none> and password file <none>
I20260926 03:47:17.749037 15698 fs_manager.cc:359] Metadata directory not provided
I20260926 03:47:17.749166 15698 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:47:17.755368 15698 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260926 03:47:17.758404 15713 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:47:17.759912 15698 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 03:47:17.760054 15698 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/minicluster-data/master-0/data,/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/minicluster-data/master-0/wal
uuid: "fd19af12fdda4aa9be763b660e4611c8"
format_stamp: "Formatted at 2026-09-26 03:47:11 on dist-test-slave-n2th"
server_key: "a8703d5d0686fe8f2af4d77f386693cf"
server_key_iv: "571485da51254437ea71931d7e4ab0d3"
server_key_version: "encryptionkey@0"
I20260926 03:47:17.761114 15698 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:47:17.778191 15698 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:47:17.782558 15698 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:47:17.782909 15698 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:47:17.800902 15698 rpc_server.cc:307] RPC server started. Bound to: 127.0.194.254:34561
I20260926 03:47:17.800907 15764 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.194.254:34561 every 8 connection(s)
I20260926 03:47:17.802536 15698 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/minicluster-data/master-0/data/info.pb
I20260926 03:47:17.809754 15765 sys_catalog.cc:263] Verifying existing consensus state
I20260926 03:47:17.812006 779 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu as pid 15698
I20260926 03:47:17.813040 779 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
/tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.194.193:38899
--local_ip_for_outbound_sockets=127.0.194.193
--tserver_master_addrs=127.0.194.254:34561
--webserver_port=45389
--webserver_interface=127.0.194.193
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/krb5kdc/kudu_127.0.194.193.keytab
--principal=kudu/127.0.194.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.194.212:45229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 03:47:17.813946 15765 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd19af12fdda4aa9be763b660e4611c8: Bootstrap starting.
I20260926 03:47:17.837658 15765 log.cc:824] T 00000000000000000000000000000000 P fd19af12fdda4aa9be763b660e4611c8: Log is configured to *not* fsync() on all Append() calls
I20260926 03:47:17.855967 15765 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd19af12fdda4aa9be763b660e4611c8: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 03:47:17.856570 15765 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd19af12fdda4aa9be763b660e4611c8: Bootstrap complete.
I20260926 03:47:17.866110 15765 raft_consensus.cc:348] T 00000000000000000000000000000000 P fd19af12fdda4aa9be763b660e4611c8 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd19af12fdda4aa9be763b660e4611c8" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 34561 } }
I20260926 03:47:17.867209 15765 raft_consensus.cc:729] T 00000000000000000000000000000000 P fd19af12fdda4aa9be763b660e4611c8 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd19af12fdda4aa9be763b660e4611c8, State: Initialized, Role: FOLLOWER
I20260926 03:47:17.867904 15765 consensus_queue.cc:260] T 00000000000000000000000000000000 P fd19af12fdda4aa9be763b660e4611c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd19af12fdda4aa9be763b660e4611c8" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 34561 } }
I20260926 03:47:17.868145 15765 raft_consensus.cc:388] T 00000000000000000000000000000000 P fd19af12fdda4aa9be763b660e4611c8 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:47:17.868270 15765 raft_consensus.cc:482] T 00000000000000000000000000000000 P fd19af12fdda4aa9be763b660e4611c8 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:47:17.868445 15765 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fd19af12fdda4aa9be763b660e4611c8 [term 2 FOLLOWER]: Advancing to term 3
I20260926 03:47:17.872534 15765 raft_consensus.cc:504] T 00000000000000000000000000000000 P fd19af12fdda4aa9be763b660e4611c8 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd19af12fdda4aa9be763b660e4611c8" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 34561 } }
I20260926 03:47:17.873055 15765 leader_election.cc:302] T 00000000000000000000000000000000 P fd19af12fdda4aa9be763b660e4611c8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fd19af12fdda4aa9be763b660e4611c8; no voters:
I20260926 03:47:17.873694 15765 leader_election.cc:288] T 00000000000000000000000000000000 P fd19af12fdda4aa9be763b660e4611c8 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260926 03:47:17.873941 15769 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fd19af12fdda4aa9be763b660e4611c8 [term 3 FOLLOWER]: Leader election won for term 3
I20260926 03:47:17.875043 15769 raft_consensus.cc:686] T 00000000000000000000000000000000 P fd19af12fdda4aa9be763b660e4611c8 [term 3 LEADER]: Becoming Leader. State: Replica: fd19af12fdda4aa9be763b660e4611c8, State: Running, Role: LEADER
I20260926 03:47:17.875625 15769 consensus_queue.cc:237] T 00000000000000000000000000000000 P fd19af12fdda4aa9be763b660e4611c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd19af12fdda4aa9be763b660e4611c8" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 34561 } }
I20260926 03:47:17.876034 15765 sys_catalog.cc:564] T 00000000000000000000000000000000 P fd19af12fdda4aa9be763b660e4611c8 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:47:17.878196 15771 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd19af12fdda4aa9be763b660e4611c8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fd19af12fdda4aa9be763b660e4611c8. Latest consensus state: current_term: 3 leader_uuid: "fd19af12fdda4aa9be763b660e4611c8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd19af12fdda4aa9be763b660e4611c8" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 34561 } } }
I20260926 03:47:17.878229 15770 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd19af12fdda4aa9be763b660e4611c8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "fd19af12fdda4aa9be763b660e4611c8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd19af12fdda4aa9be763b660e4611c8" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 34561 } } }
I20260926 03:47:17.878605 15771 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd19af12fdda4aa9be763b660e4611c8 [sys.catalog]: This master's current role is: LEADER
I20260926 03:47:17.878648 15770 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd19af12fdda4aa9be763b660e4611c8 [sys.catalog]: This master's current role is: LEADER
I20260926 03:47:17.890816 15774 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:47:17.900322 15774 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:47:17.901854 15774 catalog_manager.cc:1259] Loaded cluster ID: 043e28fe1f644dc78387fad6417f56fa
I20260926 03:47:17.901971 15774 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260926 03:47:17.904162 15774 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f4adb4139ad google::LogMessage::Fail() at ??:0
@ 0x7f4adb41688a google::LogMessage::SendToLog() at ??:0
@ 0x7f4adb4134de google::LogMessage::Flush() at ??:0
@ 0x7f4adb414a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f4af0e46512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f4af0e43976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f4af0ecb610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f4af0ecb439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f4ade05f345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f4ade06291c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
I20260926 03:47:17.948371 15788 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
@ 0x7f4ade062799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f4ade03deaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f4ae13176db start_thread at ??:0
@ 0x7f4ada2ad61f clone at ??:0
W20260926 03:47:18.228098 15767 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:47:18.228411 15767 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:47:18.228569 15767 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:47:18.236016 15767 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:47:18.236150 15767 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:47:18.236326 15767 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.194.193
I20260926 03:47:18.246567 15767 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.194.212:45229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/krb5kdc/kudu_127.0.194.193.keytab
--principal=kudu/127.0.194.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.194.193:38899
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.194.193
--webserver_port=45389
--tserver_master_addrs=127.0.194.254:34561
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.194.193
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:47:18.247990 15767 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:47:18.249364 15767 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:47:18.262795 15809 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:47:18.262903 15807 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:47:18.262851 15806 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:47:18.268388 15767 server_base.cc:1034] running on GCE node
I20260926 03:47:18.269037 15767 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:47:18.269977 15767 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:47:18.271173 15767 hybrid_clock.cc:648] HybridClock initialized: now 1790394438271137 us; error 42 us; skew 500 ppm
I20260926 03:47:18.271471 15767 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:47:18 dist-test-slave-n2th krb5kdc[14616](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394438, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:47:18.274785 15767 init.cc:377] Logged in from keytab as kudu/127.0.194.193@KRBTEST.COM (short username kudu)
I20260926 03:47:18.276371 15767 webserver.cc:460] Webserver started at http://127.0.194.193:45389/ using document root <none> and password file <none>
I20260926 03:47:18.277077 15767 fs_manager.cc:359] Metadata directory not provided
I20260926 03:47:18.277179 15767 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:47:18.282598 15767 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260926 03:47:18.287747 15816 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:47:18.289207 15767 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 03:47:18.289350 15767 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/minicluster-data/ts-0/wal
uuid: "26e7194486b140e2b7434d73bbe399e6"
format_stamp: "Formatted at 2026-09-26 03:47:12 on dist-test-slave-n2th"
server_key: "00e3ccacf58a1416f41a36bf1468fda4"
server_key_iv: "68edea37252c95df0e14a621e3c1f2fb"
server_key_version: "encryptionkey@0"
I20260926 03:47:18.290324 15767 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:47:18.334807 15767 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:47:18.339529 15767 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:47:18.339772 15767 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:47:18.340744 15767 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:47:18.342573 15823 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260926 03:47:18.348847 15767 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260926 03:47:18.348973 15767 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.007s user 0.001s sys 0.000s
I20260926 03:47:18.349081 15767 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260926 03:47:18.352360 15767 ts_tablet_manager.cc:610] Registered 1 tablets
I20260926 03:47:18.352451 15767 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.000s sys 0.002s
I20260926 03:47:18.352845 15823 tablet_bootstrap.cc:492] T 6bfdc439a5194d7eb3a2cd848d9fbd39 P 26e7194486b140e2b7434d73bbe399e6: Bootstrap starting.
I20260926 03:47:18.384513 15823 log.cc:824] T 6bfdc439a5194d7eb3a2cd848d9fbd39 P 26e7194486b140e2b7434d73bbe399e6: Log is configured to *not* fsync() on all Append() calls
I20260926 03:47:18.398976 15823 tablet_bootstrap.cc:492] T 6bfdc439a5194d7eb3a2cd848d9fbd39 P 26e7194486b140e2b7434d73bbe399e6: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 03:47:18.399240 15767 rpc_server.cc:307] RPC server started. Bound to: 127.0.194.193:38899
I20260926 03:47:18.399261 15930 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.194.193:38899 every 8 connection(s)
I20260926 03:47:18.399657 15823 tablet_bootstrap.cc:492] T 6bfdc439a5194d7eb3a2cd848d9fbd39 P 26e7194486b140e2b7434d73bbe399e6: Bootstrap complete.
I20260926 03:47:18.400640 15767 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/minicluster-data/ts-0/data/info.pb
I20260926 03:47:18.401036 15823 ts_tablet_manager.cc:1397] T 6bfdc439a5194d7eb3a2cd848d9fbd39 P 26e7194486b140e2b7434d73bbe399e6: Time spent bootstrapping tablet: real 0.049s user 0.029s sys 0.014s
I20260926 03:47:18.405215 779 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu as pid 15767
I20260926 03:47:18.405905 779 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
/tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.194.194:34653
--local_ip_for_outbound_sockets=127.0.194.194
--tserver_master_addrs=127.0.194.254:34561
--webserver_port=38561
--webserver_interface=127.0.194.194
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/krb5kdc/kudu_127.0.194.194.keytab
--principal=kudu/127.0.194.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.194.212:45229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 03:47:18.406903 15933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:18.404080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:34561 (local address 127.0.194.193:59429)
0926 03:47:18.405032 (+ 952us) negotiation.cc:107] Waiting for socket to connect
0926 03:47:18.405487 (+ 455us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.254:34561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":620,"thread_start_us":389,"threads_started":1}
W20260926 03:47:18.407727 15931 heartbeater.cc:644] Failed to heartbeat to 127.0.194.254:34561 (0 consecutive failures): Network error: Failed to ping master at 127.0.194.254:34561: Client connection negotiation failed: client connection to 127.0.194.254:34561: connect: Connection refused (error 111)
I20260926 03:47:18.409030 15933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:18.408523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:34561 (local address 127.0.194.193:46847)
0926 03:47:18.408745 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0926 03:47:18.408856 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.254:34561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260926 03:47:18.410305 15933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:18.409792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:34561 (local address 127.0.194.193:34583)
0926 03:47:18.409991 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0926 03:47:18.410104 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.254:34561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260926 03:47:18.410650 15931 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 03:47:18.410813 15823 raft_consensus.cc:348] T 6bfdc439a5194d7eb3a2cd848d9fbd39 P 26e7194486b140e2b7434d73bbe399e6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca26e0bdcc7c4021a1ae2b833aa2695f" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 34653 } } peers { permanent_uuid: "453de609dea34f9eb8f70055f5398b7e" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 33219 } } peers { permanent_uuid: "26e7194486b140e2b7434d73bbe399e6" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 38899 } }
I20260926 03:47:18.412061 15823 raft_consensus.cc:729] T 6bfdc439a5194d7eb3a2cd848d9fbd39 P 26e7194486b140e2b7434d73bbe399e6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26e7194486b140e2b7434d73bbe399e6, State: Initialized, Role: FOLLOWER
I20260926 03:47:18.412734 15823 consensus_queue.cc:260] T 6bfdc439a5194d7eb3a2cd848d9fbd39 P 26e7194486b140e2b7434d73bbe399e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca26e0bdcc7c4021a1ae2b833aa2695f" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 34653 } } peers { permanent_uuid: "453de609dea34f9eb8f70055f5398b7e" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 33219 } } peers { permanent_uuid: "26e7194486b140e2b7434d73bbe399e6" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 38899 } }
I20260926 03:47:18.414543 15823 ts_tablet_manager.cc:1428] T 6bfdc439a5194d7eb3a2cd848d9fbd39 P 26e7194486b140e2b7434d73bbe399e6: Time spent starting tablet: real 0.013s user 0.014s sys 0.001s
I20260926 03:47:18.414567 15933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:18.414106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:34561 (local address 127.0.194.193:53011)
0926 03:47:18.414282 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0926 03:47:18.414391 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.254:34561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260926 03:47:18.415722 15933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:18.415275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:34561 (local address 127.0.194.193:49925)
0926 03:47:18.415461 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0926 03:47:18.415554 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.254:34561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260926 03:47:18.750689 15934 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:47:18.751012 15934 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:47:18.751200 15934 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:47:18.758811 15934 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:47:18.758944 15934 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:47:18.759143 15934 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.194.194
I20260926 03:47:18.769148 15934 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.194.212:45229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/krb5kdc/kudu_127.0.194.194.keytab
--principal=kudu/127.0.194.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.194.194:34653
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.194.194
--webserver_port=38561
--tserver_master_addrs=127.0.194.254:34561
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.194.194
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:47:18.770556 15934 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:47:18.771968 15934 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:47:18.785264 15940 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:47:18.785347 15943 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:47:18.785323 15941 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:47:18.786212 15934 server_base.cc:1034] running on GCE node
I20260926 03:47:18.786958 15934 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:47:18.787994 15934 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:47:18.789223 15934 hybrid_clock.cc:648] HybridClock initialized: now 1790394438789168 us; error 53 us; skew 500 ppm
I20260926 03:47:18.789556 15934 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:47:18 dist-test-slave-n2th krb5kdc[14616](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394438, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:47:18.792835 15934 init.cc:377] Logged in from keytab as kudu/127.0.194.194@KRBTEST.COM (short username kudu)
I20260926 03:47:18.794376 15934 webserver.cc:460] Webserver started at http://127.0.194.194:38561/ using document root <none> and password file <none>
I20260926 03:47:18.795171 15934 fs_manager.cc:359] Metadata directory not provided
I20260926 03:47:18.795284 15934 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:47:18.800838 15934 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.001s
I20260926 03:47:18.803797 15950 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:47:18.805475 15934 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 03:47:18.805629 15934 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/minicluster-data/ts-1/wal
uuid: "ca26e0bdcc7c4021a1ae2b833aa2695f"
format_stamp: "Formatted at 2026-09-26 03:47:13 on dist-test-slave-n2th"
server_key: "df7b7109cc6eb457606f5360d9d6bff3"
server_key_iv: "70ac76747ff308b396c2a02aa945d2ba"
server_key_version: "encryptionkey@0"
I20260926 03:47:18.806638 15934 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:47:18.831360 15934 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:47:18.835451 15934 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:47:18.835687 15934 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:47:18.836647 15934 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:47:18.838063 15934 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:47:18.838155 15934 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:47:18.838239 15934 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:47:18.838302 15934 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:47:18.873477 15934 rpc_server.cc:307] RPC server started. Bound to: 127.0.194.194:34653
I20260926 03:47:18.873534 16062 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.194.194:34653 every 8 connection(s)
I20260926 03:47:18.875051 15934 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/minicluster-data/ts-1/data/info.pb
I20260926 03:47:18.880030 16065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:18.878061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:34561 (local address 127.0.194.194:45855)
0926 03:47:18.878891 (+ 830us) negotiation.cc:107] Waiting for socket to connect
0926 03:47:18.879152 (+ 261us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.254:34561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":288,"threads_started":1}
W20260926 03:47:18.880760 16063 heartbeater.cc:644] Failed to heartbeat to 127.0.194.254:34561 (0 consecutive failures): Network error: Failed to ping master at 127.0.194.254:34561: Client connection negotiation failed: client connection to 127.0.194.254:34561: connect: Connection refused (error 111)
I20260926 03:47:18.881626 16065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:18.881297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:34561 (local address 127.0.194.194:38917)
0926 03:47:18.881443 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0926 03:47:18.881516 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.254:34561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260926 03:47:18.882212 779 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu as pid 15934
I20260926 03:47:18.882578 16065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:18.882252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:34561 (local address 127.0.194.194:58409)
0926 03:47:18.882398 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0926 03:47:18.882472 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.254:34561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260926 03:47:18.882913 16063 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 03:47:18.882933 779 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
/tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.194.195:33219
--local_ip_for_outbound_sockets=127.0.194.195
--tserver_master_addrs=127.0.194.254:34561
--webserver_port=33033
--webserver_interface=127.0.194.195
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/krb5kdc/kudu_127.0.194.195.keytab
--principal=kudu/127.0.194.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.194.212:45229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:47:19.213225 16066 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:47:19.213526 16066 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:47:19.213708 16066 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:47:19.220993 16066 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:47:19.221123 16066 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:47:19.221320 16066 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.194.195
I20260926 03:47:19.231216 16066 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.194.212:45229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/krb5kdc/kudu_127.0.194.195.keytab
--principal=kudu/127.0.194.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.194.195:33219
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.194.195
--webserver_port=33033
--tserver_master_addrs=127.0.194.254:34561
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.194.195
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:47:19.232595 16066 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:47:19.233944 16066 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:47:19.247349 16072 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:47:19.247399 16074 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:47:19.247355 16071 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:47:19.248376 16066 server_base.cc:1034] running on GCE node
I20260926 03:47:19.249025 16066 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:47:19.250074 16066 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:47:19.251296 16066 hybrid_clock.cc:648] HybridClock initialized: now 1790394439251251 us; error 49 us; skew 500 ppm
I20260926 03:47:19.251631 16066 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:47:19 dist-test-slave-n2th krb5kdc[14616](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394439, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:47:19.254932 16066 init.cc:377] Logged in from keytab as kudu/127.0.194.195@KRBTEST.COM (short username kudu)
I20260926 03:47:19.256459 16066 webserver.cc:460] Webserver started at http://127.0.194.195:33033/ using document root <none> and password file <none>
I20260926 03:47:19.257184 16066 fs_manager.cc:359] Metadata directory not provided
I20260926 03:47:19.257329 16066 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:47:19.262979 16066 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260926 03:47:19.265884 16081 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:47:19.267414 16066 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 03:47:19.267601 16066 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/minicluster-data/ts-2/wal
uuid: "453de609dea34f9eb8f70055f5398b7e"
format_stamp: "Formatted at 2026-09-26 03:47:13 on dist-test-slave-n2th"
server_key: "5bcb29a21950bef073643bfa6cba164f"
server_key_iv: "da0f90276eff00af416a0a415022a12d"
server_key_version: "encryptionkey@0"
I20260926 03:47:19.268606 16066 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:47:19.292332 16066 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:47:19.296727 16066 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:47:19.297015 16066 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:47:19.298079 16066 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:47:19.299536 16066 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:47:19.299669 16066 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:47:19.299804 16066 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:47:19.299889 16066 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:47:19.334172 16066 rpc_server.cc:307] RPC server started. Bound to: 127.0.194.195:33219
I20260926 03:47:19.334231 16193 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.194.195:33219 every 8 connection(s)
I20260926 03:47:19.335744 16066 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790394221564982-779-0/minicluster-data/ts-2/data/info.pb
I20260926 03:47:19.340775 16196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:19.338835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:34561 (local address 127.0.194.195:42093)
0926 03:47:19.339560 (+ 725us) negotiation.cc:107] Waiting for socket to connect
0926 03:47:19.339819 (+ 259us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.254:34561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":258,"threads_started":1}
W20260926 03:47:19.341423 16194 heartbeater.cc:644] Failed to heartbeat to 127.0.194.254:34561 (0 consecutive failures): Network error: Failed to ping master at 127.0.194.254:34561: Client connection negotiation failed: client connection to 127.0.194.254:34561: connect: Connection refused (error 111)
I20260926 03:47:19.342223 16196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:19.341923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:34561 (local address 127.0.194.195:33001)
0926 03:47:19.342057 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0926 03:47:19.342121 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.254:34561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260926 03:47:19.343034 16196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:19.342710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:34561 (local address 127.0.194.195:53613)
0926 03:47:19.342851 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0926 03:47:19.342930 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.254:34561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260926 03:47:19.343297 16194 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 03:47:19.346004 779 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu as pid 16066
I20260926 03:47:19.417337 16198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:19.416563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:34561 (local address 127.0.194.193:44865)
0926 03:47:19.417024 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0926 03:47:19.417127 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.254:34561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":231,"threads_started":1}
I20260926 03:47:19.884336 16199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:19.883472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:34561 (local address 127.0.194.194:45453)
0926 03:47:19.884011 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0926 03:47:19.884101 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.254:34561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":273,"threads_started":1}
I20260926 03:47:20.227744 16200 raft_consensus.cc:482] T 6bfdc439a5194d7eb3a2cd848d9fbd39 P 26e7194486b140e2b7434d73bbe399e6 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 03:47:20.227991 16200 raft_consensus.cc:504] T 6bfdc439a5194d7eb3a2cd848d9fbd39 P 26e7194486b140e2b7434d73bbe399e6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca26e0bdcc7c4021a1ae2b833aa2695f" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 34653 } } peers { permanent_uuid: "453de609dea34f9eb8f70055f5398b7e" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 33219 } } peers { permanent_uuid: "26e7194486b140e2b7434d73bbe399e6" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 38899 } }
I20260926 03:47:20.229878 16200 leader_election.cc:288] T 6bfdc439a5194d7eb3a2cd848d9fbd39 P 26e7194486b140e2b7434d73bbe399e6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ca26e0bdcc7c4021a1ae2b833aa2695f (127.0.194.194:34653), 453de609dea34f9eb8f70055f5398b7e (127.0.194.195:33219)
Sep 26 03:47:20 dist-test-slave-n2th krb5kdc[14616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394438, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.193@KRBTEST.COM for kudu/127.0.194.195@KRBTEST.COM
Sep 26 03:47:20 dist-test-slave-n2th krb5kdc[14616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394438, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.193@KRBTEST.COM for kudu/127.0.194.194@KRBTEST.COM
I20260926 03:47:20.263581 16203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:20.230450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.193:52101 (local address 127.0.194.194:34653)
0926 03:47:20.241045 (+ 10595us) server_negotiation.cc:206] Beginning negotiation
0926 03:47:20.241073 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0926 03:47:20.241939 (+ 866us) server_negotiation.cc:407] Connection header received
0926 03:47:20.244493 (+ 2554us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:47:20.244503 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:47:20.244889 (+ 386us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:47:20.245246 (+ 357us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:47:20.247544 (+ 2298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:47:20.248741 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:47:20.250607 (+ 1866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:47:20.251333 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:47:20.251555 (+ 222us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:47:20.255954 (+ 4399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:47:20.255962 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:47:20.255974 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:47:20.256013 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:47:20.259589 (+ 3576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:47:20.260857 (+ 1268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:47:20.260864 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:47:20.260869 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:47:20.261022 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:47:20.261429 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:47:20.261433 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:47:20.261435 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:47:20.261972 (+ 537us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:47:20.262159 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0926 03:47:20.262756 (+ 597us) server_negotiation.cc:299] Negotiation successful
0926 03:47:20.263136 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10320,"thread_start_us":10089,"threads_started":1}
I20260926 03:47:20.263594 16202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:20.230201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.194:34653 (local address 127.0.194.193:52101)
0926 03:47:20.241606 (+ 11405us) negotiation.cc:107] Waiting for socket to connect
0926 03:47:20.241769 (+ 163us) client_negotiation.cc:174] Beginning negotiation
0926 03:47:20.242416 (+ 647us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:47:20.245428 (+ 3012us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:47:20.245440 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:47:20.246432 (+ 992us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:47:20.247327 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:47:20.247349 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:47:20.248942 (+ 1593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:47:20.248946 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:47:20.250458 (+ 1512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:47:20.250471 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:47:20.251548 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:47:20.251555 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:47:20.251838 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:47:20.252790 (+ 952us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:47:20.252813 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:47:20.255677 (+ 2864us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:47:20.260389 (+ 4712us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:47:20.260392 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:47:20.260394 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:47:20.260708 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:47:20.261179 (+ 471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:47:20.261182 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:47:20.261185 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:47:20.261306 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:47:20.262106 (+ 800us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:47:20.262110 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:47:20.262276 (+ 166us) client_negotiation.cc:769] Sending connection context
0926 03:47:20.262480 (+ 204us) client_negotiation.cc:240] Negotiation successful
0926 03:47:20.263103 (+ 623us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10857,"mutex_wait_us":145,"spinlock_wait_cycles":19712,"thread_start_us":11106,"threads_started":1}
I20260926 03:47:20.263664 16201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:20.230206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.195:33219 (local address 127.0.194.193:42281)
0926 03:47:20.241763 (+ 11557us) negotiation.cc:107] Waiting for socket to connect
0926 03:47:20.241786 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 03:47:20.242422 (+ 636us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:47:20.245428 (+ 3006us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:47:20.245440 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:47:20.246432 (+ 992us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:47:20.247328 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:47:20.247349 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:47:20.249010 (+ 1661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:47:20.249015 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:47:20.250461 (+ 1446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:47:20.250474 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:47:20.251632 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:47:20.251644 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:47:20.251906 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:47:20.252717 (+ 811us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:47:20.252750 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:47:20.255410 (+ 2660us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:47:20.259773 (+ 4363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:47:20.259781 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:47:20.259795 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:47:20.260272 (+ 477us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:47:20.260812 (+ 540us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:47:20.260817 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:47:20.260821 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:47:20.261031 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:47:20.261861 (+ 830us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:47:20.261870 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:47:20.262195 (+ 325us) client_negotiation.cc:769] Sending connection context
0926 03:47:20.262483 (+ 288us) client_negotiation.cc:240] Negotiation successful
0926 03:47:20.263262 (+ 779us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11312,"mutex_wait_us":566,"spinlock_wait_cycles":68608,"thread_start_us":10946,"threads_started":1}
I20260926 03:47:20.264657 16204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:20.230690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.193:42281 (local address 127.0.194.195:33219)
0926 03:47:20.241760 (+ 11070us) server_negotiation.cc:206] Beginning negotiation
0926 03:47:20.241780 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0926 03:47:20.241941 (+ 161us) server_negotiation.cc:407] Connection header received
0926 03:47:20.244434 (+ 2493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:47:20.244444 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:47:20.244889 (+ 445us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:47:20.245249 (+ 360us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:47:20.247552 (+ 2303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:47:20.248821 (+ 1269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:47:20.250638 (+ 1817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:47:20.251393 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:47:20.251615 (+ 222us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:47:20.255732 (+ 4117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:47:20.255740 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:47:20.255757 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:47:20.255796 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:47:20.259584 (+ 3788us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:47:20.260431 (+ 847us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:47:20.260442 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:47:20.260451 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:47:20.260679 (+ 228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:47:20.261155 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:47:20.261160 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:47:20.261168 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:47:20.261710 (+ 542us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:47:20.261927 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0926 03:47:20.262708 (+ 781us) server_negotiation.cc:299] Negotiation successful
0926 03:47:20.263123 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9512,"thread_start_us":9323,"threads_started":1}
W20260926 03:47:20.267024 15817 leader_election.cc:341] T 6bfdc439a5194d7eb3a2cd848d9fbd39 P 26e7194486b140e2b7434d73bbe399e6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ca26e0bdcc7c4021a1ae2b833aa2695f (127.0.194.194:34653): Not found: Tablet not found: 6bfdc439a5194d7eb3a2cd848d9fbd39
W20260926 03:47:20.267380 15820 leader_election.cc:341] T 6bfdc439a5194d7eb3a2cd848d9fbd39 P 26e7194486b140e2b7434d73bbe399e6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 453de609dea34f9eb8f70055f5398b7e (127.0.194.195:33219): Not found: Tablet not found: 6bfdc439a5194d7eb3a2cd848d9fbd39
I20260926 03:47:20.267531 15820 leader_election.cc:302] T 6bfdc439a5194d7eb3a2cd848d9fbd39 P 26e7194486b140e2b7434d73bbe399e6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26e7194486b140e2b7434d73bbe399e6; no voters: 453de609dea34f9eb8f70055f5398b7e, ca26e0bdcc7c4021a1ae2b833aa2695f
I20260926 03:47:20.267937 16200 raft_consensus.cc:2726] T 6bfdc439a5194d7eb3a2cd848d9fbd39 P 26e7194486b140e2b7434d73bbe399e6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 03:47:20.344858 16205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:20.343970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:34561 (local address 127.0.194.195:53085)
0926 03:47:20.344466 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0926 03:47:20.344621 (+ 155us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.254:34561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":236,"threads_started":1}
I20260926 03:47:20.418813 16202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:20.418304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:34561 (local address 127.0.194.193:45987)
0926 03:47:20.418503 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0926 03:47:20.418616 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.254:34561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260926 03:47:20.885974 16206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:20.885277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:34561 (local address 127.0.194.194:35883)
0926 03:47:20.885718 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0926 03:47:20.885798 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.254:34561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":244,"threads_started":1}
I20260926 03:47:21.346635 16207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:21.345803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:34561 (local address 127.0.194.195:59953)
0926 03:47:21.346346 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0926 03:47:21.346462 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.254:34561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":292,"threads_started":1}
I20260926 03:47:21.420543 16208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:21.419720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:34561 (local address 127.0.194.193:57237)
0926 03:47:21.420262 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0926 03:47:21.420362 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.254:34561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":277,"threads_started":1}
I20260926 03:47:21.887900 16209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:21.886964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:34561 (local address 127.0.194.194:39757)
0926 03:47:21.887515 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0926 03:47:21.887628 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.254:34561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":273,"threads_started":1}
I20260926 03:47:22.208770 16210 raft_consensus.cc:482] T 6bfdc439a5194d7eb3a2cd848d9fbd39 P 26e7194486b140e2b7434d73bbe399e6 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 03:47:22.209005 16210 raft_consensus.cc:504] T 6bfdc439a5194d7eb3a2cd848d9fbd39 P 26e7194486b140e2b7434d73bbe399e6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca26e0bdcc7c4021a1ae2b833aa2695f" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 34653 } } peers { permanent_uuid: "453de609dea34f9eb8f70055f5398b7e" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 33219 } } peers { permanent_uuid: "26e7194486b140e2b7434d73bbe399e6" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 38899 } }
I20260926 03:47:22.209731 16210 leader_election.cc:288] T 6bfdc439a5194d7eb3a2cd848d9fbd39 P 26e7194486b140e2b7434d73bbe399e6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ca26e0bdcc7c4021a1ae2b833aa2695f (127.0.194.194:34653), 453de609dea34f9eb8f70055f5398b7e (127.0.194.195:33219)
W20260926 03:47:22.210943 15817 leader_election.cc:341] T 6bfdc439a5194d7eb3a2cd848d9fbd39 P 26e7194486b140e2b7434d73bbe399e6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ca26e0bdcc7c4021a1ae2b833aa2695f (127.0.194.194:34653): Not found: Tablet not found: 6bfdc439a5194d7eb3a2cd848d9fbd39
W20260926 03:47:22.211199 15820 leader_election.cc:341] T 6bfdc439a5194d7eb3a2cd848d9fbd39 P 26e7194486b140e2b7434d73bbe399e6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 453de609dea34f9eb8f70055f5398b7e (127.0.194.195:33219): Not found: Tablet not found: 6bfdc439a5194d7eb3a2cd848d9fbd39
I20260926 03:47:22.211375 15820 leader_election.cc:302] T 6bfdc439a5194d7eb3a2cd848d9fbd39 P 26e7194486b140e2b7434d73bbe399e6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26e7194486b140e2b7434d73bbe399e6; no voters: 453de609dea34f9eb8f70055f5398b7e, ca26e0bdcc7c4021a1ae2b833aa2695f
I20260926 03:47:22.211625 16210 raft_consensus.cc:2726] T 6bfdc439a5194d7eb3a2cd848d9fbd39 P 26e7194486b140e2b7434d73bbe399e6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 03:47:22.348351 16211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:22.347653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:34561 (local address 127.0.194.195:53409)
0926 03:47:22.348102 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0926 03:47:22.348186 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.254:34561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":231,"threads_started":1}
I20260926 03:47:22.422248 16212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:22.421475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:34561 (local address 127.0.194.193:43859)
0926 03:47:22.421936 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0926 03:47:22.422064 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.254:34561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":212,"threads_started":1}
I20260926 03:47:22.889559 16213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:22.888848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:34561 (local address 127.0.194.194:35565)
0926 03:47:22.889305 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0926 03:47:22.889386 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.254:34561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":227,"threads_started":1}
I20260926 03:47:23.349890 16214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:23.349187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:34561 (local address 127.0.194.195:54501)
0926 03:47:23.349586 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0926 03:47:23.349662 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.254:34561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":200,"threads_started":1}
I20260926 03:47:23.424211 16215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:23.423487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:34561 (local address 127.0.194.193:36727)
0926 03:47:23.423938 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0926 03:47:23.424024 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.254:34561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":236,"threads_started":1}
I20260926 03:47:23.891117 16216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:23.890391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:34561 (local address 127.0.194.194:35525)
0926 03:47:23.890875 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0926 03:47:23.890956 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.254:34561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":225,"threads_started":1}
I20260926 03:47:24.351675 16217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:24.350895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:34561 (local address 127.0.194.195:49353)
0926 03:47:24.351400 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0926 03:47:24.351508 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.254:34561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":248,"threads_started":1}
I20260926 03:47:24.425915 16218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:24.425200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:34561 (local address 127.0.194.193:42779)
0926 03:47:24.425664 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0926 03:47:24.425756 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.254:34561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":261,"threads_started":1}
I20260926 03:47:24.761214 16219 raft_consensus.cc:482] T 6bfdc439a5194d7eb3a2cd848d9fbd39 P 26e7194486b140e2b7434d73bbe399e6 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 03:47:24.761461 16219 raft_consensus.cc:504] T 6bfdc439a5194d7eb3a2cd848d9fbd39 P 26e7194486b140e2b7434d73bbe399e6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca26e0bdcc7c4021a1ae2b833aa2695f" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 34653 } } peers { permanent_uuid: "453de609dea34f9eb8f70055f5398b7e" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 33219 } } peers { permanent_uuid: "26e7194486b140e2b7434d73bbe399e6" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 38899 } }
I20260926 03:47:24.762099 16219 leader_election.cc:288] T 6bfdc439a5194d7eb3a2cd848d9fbd39 P 26e7194486b140e2b7434d73bbe399e6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ca26e0bdcc7c4021a1ae2b833aa2695f (127.0.194.194:34653), 453de609dea34f9eb8f70055f5398b7e (127.0.194.195:33219)
W20260926 03:47:24.763187 15820 leader_election.cc:341] T 6bfdc439a5194d7eb3a2cd848d9fbd39 P 26e7194486b140e2b7434d73bbe399e6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 453de609dea34f9eb8f70055f5398b7e (127.0.194.195:33219): Not found: Tablet not found: 6bfdc439a5194d7eb3a2cd848d9fbd39
W20260926 03:47:24.763398 15817 leader_election.cc:341] T 6bfdc439a5194d7eb3a2cd848d9fbd39 P 26e7194486b140e2b7434d73bbe399e6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ca26e0bdcc7c4021a1ae2b833aa2695f (127.0.194.194:34653): Not found: Tablet not found: 6bfdc439a5194d7eb3a2cd848d9fbd39
I20260926 03:47:24.763495 15817 leader_election.cc:302] T 6bfdc439a5194d7eb3a2cd848d9fbd39 P 26e7194486b140e2b7434d73bbe399e6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26e7194486b140e2b7434d73bbe399e6; no voters: 453de609dea34f9eb8f70055f5398b7e, ca26e0bdcc7c4021a1ae2b833aa2695f
I20260926 03:47:24.763727 16219 raft_consensus.cc:2726] T 6bfdc439a5194d7eb3a2cd848d9fbd39 P 26e7194486b140e2b7434d73bbe399e6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 03:47:24.893107 16220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:24.892182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:34561 (local address 127.0.194.194:47219)
0926 03:47:24.892782 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0926 03:47:24.892897 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.254:34561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":307,"threads_started":1}
I20260926 03:47:25.353498 16221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:25.352690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:34561 (local address 127.0.194.195:34439)
0926 03:47:25.353169 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0926 03:47:25.353257 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.254:34561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":252,"threads_started":1}
I20260926 03:47:25.428059 16222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:25.427027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:34561 (local address 127.0.194.193:55879)
0926 03:47:25.427632 (+ 605us) negotiation.cc:107] Waiting for socket to connect
0926 03:47:25.427769 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.254:34561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":303,"threads_started":1}
I20260926 03:47:25.895200 16223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:25.894251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:34561 (local address 127.0.194.194:56451)
0926 03:47:25.894857 (+ 606us) negotiation.cc:107] Waiting for socket to connect
0926 03:47:25.894964 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.254:34561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":240,"threads_started":1}
I20260926 03:47:26.355414 16224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:26.354551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:34561 (local address 127.0.194.195:44773)
0926 03:47:26.355126 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0926 03:47:26.355212 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.254:34561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":293,"threads_started":1}
I20260926 03:47:26.429934 16225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:26.429217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:34561 (local address 127.0.194.193:42645)
0926 03:47:26.429682 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0926 03:47:26.429778 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.254:34561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":236,"threads_started":1}
I20260926 03:47:26.896970 16226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:26.896210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:34561 (local address 127.0.194.194:41561)
0926 03:47:26.896690 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0926 03:47:26.896781 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.254:34561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":249,"threads_started":1}
I20260926 03:47:26.927227 16227 raft_consensus.cc:482] T 6bfdc439a5194d7eb3a2cd848d9fbd39 P 26e7194486b140e2b7434d73bbe399e6 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 03:47:26.927479 16227 raft_consensus.cc:504] T 6bfdc439a5194d7eb3a2cd848d9fbd39 P 26e7194486b140e2b7434d73bbe399e6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca26e0bdcc7c4021a1ae2b833aa2695f" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 34653 } } peers { permanent_uuid: "453de609dea34f9eb8f70055f5398b7e" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 33219 } } peers { permanent_uuid: "26e7194486b140e2b7434d73bbe399e6" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 38899 } }
I20260926 03:47:26.928186 16227 leader_election.cc:288] T 6bfdc439a5194d7eb3a2cd848d9fbd39 P 26e7194486b140e2b7434d73bbe399e6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ca26e0bdcc7c4021a1ae2b833aa2695f (127.0.194.194:34653), 453de609dea34f9eb8f70055f5398b7e (127.0.194.195:33219)
W20260926 03:47:26.929240 15820 leader_election.cc:341] T 6bfdc439a5194d7eb3a2cd848d9fbd39 P 26e7194486b140e2b7434d73bbe399e6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 453de609dea34f9eb8f70055f5398b7e (127.0.194.195:33219): Not found: Tablet not found: 6bfdc439a5194d7eb3a2cd848d9fbd39
W20260926 03:47:26.929443 15817 leader_election.cc:341] T 6bfdc439a5194d7eb3a2cd848d9fbd39 P 26e7194486b140e2b7434d73bbe399e6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ca26e0bdcc7c4021a1ae2b833aa2695f (127.0.194.194:34653): Not found: Tablet not found: 6bfdc439a5194d7eb3a2cd848d9fbd39
I20260926 03:47:26.929544 15817 leader_election.cc:302] T 6bfdc439a5194d7eb3a2cd848d9fbd39 P 26e7194486b140e2b7434d73bbe399e6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26e7194486b140e2b7434d73bbe399e6; no voters: 453de609dea34f9eb8f70055f5398b7e, ca26e0bdcc7c4021a1ae2b833aa2695f
I20260926 03:47:26.929773 16227 raft_consensus.cc:2726] T 6bfdc439a5194d7eb3a2cd848d9fbd39 P 26e7194486b140e2b7434d73bbe399e6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 03:47:27.357218 16228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:27.356458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:34561 (local address 127.0.194.195:35661)
0926 03:47:27.356941 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0926 03:47:27.357032 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.254:34561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":255,"threads_started":1}
I20260926 03:47:27.432000 16229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:27.431266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:34561 (local address 127.0.194.193:55745)
0926 03:47:27.431722 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0926 03:47:27.431826 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.254:34561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":234,"threads_started":1}
I20260926 03:47:27.898797 16230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:27.898018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:34561 (local address 127.0.194.194:40155)
0926 03:47:27.898491 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0926 03:47:27.898583 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.254:34561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":236,"threads_started":1}
I20260926 03:47:28.359097 16231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:28.358319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:34561 (local address 127.0.194.195:33005)
0926 03:47:28.358841 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0926 03:47:28.358943 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.254:34561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":234,"threads_started":1}
I20260926 03:47:28.433789 16232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:28.433094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:34561 (local address 127.0.194.193:35773)
0926 03:47:28.433528 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0926 03:47:28.433616 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.254:34561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":237,"threads_started":1}
I20260926 03:47:28.881903 16233 raft_consensus.cc:482] T 6bfdc439a5194d7eb3a2cd848d9fbd39 P 26e7194486b140e2b7434d73bbe399e6 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 03:47:28.882125 16233 raft_consensus.cc:504] T 6bfdc439a5194d7eb3a2cd848d9fbd39 P 26e7194486b140e2b7434d73bbe399e6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca26e0bdcc7c4021a1ae2b833aa2695f" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 34653 } } peers { permanent_uuid: "453de609dea34f9eb8f70055f5398b7e" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 33219 } } peers { permanent_uuid: "26e7194486b140e2b7434d73bbe399e6" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 38899 } }
I20260926 03:47:28.882867 16233 leader_election.cc:288] T 6bfdc439a5194d7eb3a2cd848d9fbd39 P 26e7194486b140e2b7434d73bbe399e6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ca26e0bdcc7c4021a1ae2b833aa2695f (127.0.194.194:34653), 453de609dea34f9eb8f70055f5398b7e (127.0.194.195:33219)
W20260926 03:47:28.883882 15820 leader_election.cc:341] T 6bfdc439a5194d7eb3a2cd848d9fbd39 P 26e7194486b140e2b7434d73bbe399e6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 453de609dea34f9eb8f70055f5398b7e (127.0.194.195:33219): Not found: Tablet not found: 6bfdc439a5194d7eb3a2cd848d9fbd39
W20260926 03:47:28.884051 15817 leader_election.cc:341] T 6bfdc439a5194d7eb3a2cd848d9fbd39 P 26e7194486b140e2b7434d73bbe399e6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ca26e0bdcc7c4021a1ae2b833aa2695f (127.0.194.194:34653): Not found: Tablet not found: 6bfdc439a5194d7eb3a2cd848d9fbd39
I20260926 03:47:28.884186 15817 leader_election.cc:302] T 6bfdc439a5194d7eb3a2cd848d9fbd39 P 26e7194486b140e2b7434d73bbe399e6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26e7194486b140e2b7434d73bbe399e6; no voters: 453de609dea34f9eb8f70055f5398b7e, ca26e0bdcc7c4021a1ae2b833aa2695f
I20260926 03:47:28.884428 16233 raft_consensus.cc:2726] T 6bfdc439a5194d7eb3a2cd848d9fbd39 P 26e7194486b140e2b7434d73bbe399e6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 03:47:28.900455 16234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:28.899729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:34561 (local address 127.0.194.194:32995)
0926 03:47:28.900192 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0926 03:47:28.900277 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.254:34561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":201,"threads_started":1}
I20260926 03:47:29.360910 16235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:29.360198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:34561 (local address 127.0.194.195:56721)
0926 03:47:29.360644 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0926 03:47:29.360729 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.254:34561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":228,"threads_started":1}
I20260926 03:47:29.435571 16236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:29.434834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:34561 (local address 127.0.194.193:34657)
0926 03:47:29.435299 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0926 03:47:29.435395 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.254:34561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":241,"threads_started":1}
I20260926 03:47:29.902206 16237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:29.901436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:34561 (local address 127.0.194.194:34253)
0926 03:47:29.901930 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0926 03:47:29.902022 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.254:34561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":229,"threads_started":1}
I20260926 03:47:30.362646 16238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:30.361824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:34561 (local address 127.0.194.195:36575)
0926 03:47:30.362349 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0926 03:47:30.362463 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.254:34561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":259,"threads_started":1}
I20260926 03:47:30.437150 16239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:30.436440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:34561 (local address 127.0.194.193:43551)
0926 03:47:30.436885 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0926 03:47:30.436987 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.254:34561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":216,"threads_started":1}
I20260926 03:47:30.903942 16240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:30.903202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:34561 (local address 127.0.194.194:47825)
0926 03:47:30.903669 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0926 03:47:30.903759 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.254:34561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":226,"threads_started":1}
I20260926 03:47:31.364454 16241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:31.363711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:34561 (local address 127.0.194.195:48109)
0926 03:47:31.364189 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0926 03:47:31.364271 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.254:34561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":241,"threads_started":1}
I20260926 03:47:31.438881 16242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:31.438078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:34561 (local address 127.0.194.193:43465)
0926 03:47:31.438525 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0926 03:47:31.438616 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.254:34561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":238,"threads_started":1}
I20260926 03:47:31.905818 16243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:31.905072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:34561 (local address 127.0.194.194:47737)
0926 03:47:31.905539 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0926 03:47:31.905649 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.254:34561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":238,"threads_started":1}
I20260926 03:47:32.366133 16244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:32.365413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:34561 (local address 127.0.194.195:45617)
0926 03:47:32.365860 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0926 03:47:32.365961 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.254:34561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":227,"threads_started":1}
I20260926 03:47:32.440454 16245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:32.439740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:34561 (local address 127.0.194.193:38291)
0926 03:47:32.440184 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0926 03:47:32.440266 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.254:34561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":221,"threads_started":1}
I20260926 03:47:32.907847 16246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:32.907068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:34561 (local address 127.0.194.194:46147)
0926 03:47:32.907564 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0926 03:47:32.907655 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.254:34561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":242,"threads_started":1}
I20260926 03:47:33.367874 16247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:33.367071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:34561 (local address 127.0.194.195:41413)
0926 03:47:33.367579 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0926 03:47:33.367659 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.254:34561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":264,"threads_started":1}
I20260926 03:47:33.442399 16248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:33.441672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:34561 (local address 127.0.194.193:34999)
0926 03:47:33.442119 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0926 03:47:33.442226 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.254:34561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":232,"threads_started":1}
I20260926 03:47:33.910018 16249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:33.909096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:34561 (local address 127.0.194.194:49391)
0926 03:47:33.909601 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0926 03:47:33.909797 (+ 196us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.254:34561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":257,"threads_started":1}
I20260926 03:47:34.350124 779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu pid 15767
I20260926 03:47:34.369814 16251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:34.369051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:34561 (local address 127.0.194.195:48049)
0926 03:47:34.369556 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0926 03:47:34.369660 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.254:34561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":291,"threads_started":1}
I20260926 03:47:34.372857 16250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:34.351293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45902 (local address 127.0.194.193:38899)
0926 03:47:34.351869 (+ 576us) server_negotiation.cc:206] Beginning negotiation
0926 03:47:34.351881 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 03:47:34.351916 (+ 35us) server_negotiation.cc:407] Connection header received
0926 03:47:34.351998 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:47:34.352009 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:47:34.352298 (+ 289us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:47:34.352498 (+ 200us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:47:34.354439 (+ 1941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:47:34.355501 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:47:34.356754 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:47:34.357333 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:47:34.357473 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:47:34.366713 (+ 9240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:47:34.366781 (+ 68us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:47:34.366805 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:47:34.366863 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:47:34.369730 (+ 2867us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:47:34.370587 (+ 857us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:47:34.370595 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:47:34.370597 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:47:34.370790 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:47:34.371201 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:47:34.371207 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:47:34.371211 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:47:34.371585 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:47:34.371915 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0926 03:47:34.372268 (+ 353us) server_negotiation.cc:299] Negotiation successful
0926 03:47:34.372540 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":249,"threads_started":1}
I20260926 03:47:34.374126 15926 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:47:34.471614 779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu with pid 15767
I20260926 03:47:34.490144 779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu pid 15934
I20260926 03:47:34.503424 16254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:34.490975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56966 (local address 127.0.194.194:34653)
0926 03:47:34.491453 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0926 03:47:34.491458 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:47:34.491479 (+ 21us) server_negotiation.cc:407] Connection header received
0926 03:47:34.491537 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:47:34.491541 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:47:34.491603 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:47:34.491708 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:47:34.493089 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:47:34.493885 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:47:34.495154 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:47:34.495593 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:47:34.495697 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:47:34.498395 (+ 2698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:47:34.498400 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:47:34.498403 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:47:34.498441 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:47:34.501018 (+ 2577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:47:34.501631 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:47:34.501635 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:47:34.501637 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:47:34.501768 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:47:34.502081 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:47:34.502084 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:47:34.502086 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:47:34.502358 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:47:34.502552 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0926 03:47:34.502851 (+ 299us) server_negotiation.cc:299] Negotiation successful
0926 03:47:34.503025 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":255,"threads_started":1}
I20260926 03:47:34.504145 16058 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:47:34.591596 779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu with pid 15934
I20260926 03:47:34.609967 779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu pid 16066
I20260926 03:47:34.624411 16256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:34.610759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52990 (local address 127.0.194.195:33219)
0926 03:47:34.611331 (+ 572us) server_negotiation.cc:206] Beginning negotiation
0926 03:47:34.611335 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:47:34.611350 (+ 15us) server_negotiation.cc:407] Connection header received
0926 03:47:34.611412 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:47:34.611416 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:47:34.611489 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:47:34.611616 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:47:34.613072 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:47:34.613867 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:47:34.615431 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:47:34.615872 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:47:34.616018 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:47:34.618830 (+ 2812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:47:34.618836 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:47:34.618839 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:47:34.618882 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:47:34.621552 (+ 2670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:47:34.622253 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:47:34.622260 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:47:34.622264 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:47:34.622425 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:47:34.622927 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:47:34.622931 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:47:34.622933 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:47:34.623200 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:47:34.623487 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0926 03:47:34.623797 (+ 310us) server_negotiation.cc:299] Negotiation successful
0926 03:47:34.624041 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":432,"thread_start_us":265,"threads_started":1}
I20260926 03:47:34.625185 16189 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:47:34.710319 779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu with pid 16066
2026-09-26T03:47:34Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (24474 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:47:34 dist-test-slave-n2th krb5kdc[16260](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:47:34 dist-test-slave-n2th krb5kdc[16260](info): set up 2 sockets
Sep 26 03:47:34 dist-test-slave-n2th krb5kdc[16260](info): commencing operation
krb5kdc: starting...
W20260926 03:47:35.807408 779 mini_kdc.cc:121] Time spent starting KDC: real 1.057s user 0.005s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/krb5kdc/test-user.keytab.
Sep 26 03:47:35 dist-test-slave-n2th krb5kdc[16260](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394455, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:47:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:47:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.194.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.194.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.194.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/krb5kdc/kudu_127.0.194.254.keytab.
Entry for principal kudu/127.0.194.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/krb5kdc/kudu_127.0.194.254.keytab.
I20260926 03:47:36.058843 779 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
/tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.194.254:41965
--webserver_interface=127.0.194.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/krb5kdc/kudu_127.0.194.254.keytab
--principal=kudu/127.0.194.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.194.212:43861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.194.254:41965
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:47:36.421098 16273 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:47:36.421481 16273 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:47:36.421639 16273 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:47:36.429106 16273 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260926 03:47:36.429271 16273 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260926 03:47:36.429345 16273 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260926 03:47:36.429445 16273 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:47:36.429533 16273 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:47:36.429664 16273 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:47:36.429754 16273 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:47:36.429836 16273 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:47:36.440006 16273 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.194.212:43861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.194.254:41965
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/krb5kdc/kudu_127.0.194.254.keytab
--principal=kudu/127.0.194.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.194.254:41965
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.194.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:47:36.441407 16273 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:47:36.442907 16273 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:47:36.455504 16278 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:47:36.455564 16279 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:47:36.455868 16273 server_base.cc:1034] running on GCE node
W20260926 03:47:36.455471 16281 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:47:36.456934 16273 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:47:36.458482 16273 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:47:36.459780 16273 hybrid_clock.cc:648] HybridClock initialized: now 1790394456459710 us; error 38 us; skew 500 ppm
I20260926 03:47:36.460181 16273 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:47:36 dist-test-slave-n2th krb5kdc[16260](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394456, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:47:36.463601 16273 init.cc:377] Logged in from keytab as kudu/127.0.194.254@KRBTEST.COM (short username kudu)
I20260926 03:47:36.465243 16273 webserver.cc:460] Webserver started at http://127.0.194.254:44757/ using document root <none> and password file <none>
I20260926 03:47:36.466009 16273 fs_manager.cc:359] Metadata directory not provided
I20260926 03:47:36.466150 16273 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:47:36.466485 16273 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:47:36.469669 16273 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/minicluster-data/master-0/data/instance:
uuid: "83b249b98dce4c2c86d27ac0720f4f75"
format_stamp: "Formatted at 2026-09-26 03:47:36 on dist-test-slave-n2th"
server_key: "3264ade5582dda9c739fb0e857be8854"
server_key_iv: "8092c7d8338af7094eb3e0d4524d12c5"
server_key_version: "encryptionkey@0"
I20260926 03:47:36.470515 16273 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/minicluster-data/master-0/wal/instance:
uuid: "83b249b98dce4c2c86d27ac0720f4f75"
format_stamp: "Formatted at 2026-09-26 03:47:36 on dist-test-slave-n2th"
server_key: "3264ade5582dda9c739fb0e857be8854"
server_key_iv: "8092c7d8338af7094eb3e0d4524d12c5"
server_key_version: "encryptionkey@0"
I20260926 03:47:36.476164 16273 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260926 03:47:36.479923 16288 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:47:36.481438 16273 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260926 03:47:36.481649 16273 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/minicluster-data/master-0/data,/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/minicluster-data/master-0/wal
uuid: "83b249b98dce4c2c86d27ac0720f4f75"
format_stamp: "Formatted at 2026-09-26 03:47:36 on dist-test-slave-n2th"
server_key: "3264ade5582dda9c739fb0e857be8854"
server_key_iv: "8092c7d8338af7094eb3e0d4524d12c5"
server_key_version: "encryptionkey@0"
I20260926 03:47:36.481920 16273 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:47:36.500818 16273 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:47:36.505928 16273 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:47:36.506265 16273 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:47:36.526520 16273 rpc_server.cc:307] RPC server started. Bound to: 127.0.194.254:41965
I20260926 03:47:36.526542 16339 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.194.254:41965 every 8 connection(s)
I20260926 03:47:36.528204 16273 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/minicluster-data/master-0/data/info.pb
I20260926 03:47:36.532080 16340 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:47:36.534572 779 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu as pid 16273
I20260926 03:47:36.534834 779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/minicluster-data/master-0/wal/instance
I20260926 03:47:36.535646 779 external_mini_cluster.cc:1428] Setting key 184e87cf7207f0b659b59ac27d94a27e
I20260926 03:47:36.545504 16340 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83b249b98dce4c2c86d27ac0720f4f75: Bootstrap starting.
I20260926 03:47:36.548795 16340 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 83b249b98dce4c2c86d27ac0720f4f75: Neither blocks nor log segments found. Creating new log.
I20260926 03:47:36.549832 16340 log.cc:824] T 00000000000000000000000000000000 P 83b249b98dce4c2c86d27ac0720f4f75: Log is configured to *not* fsync() on all Append() calls
Sep 26 03:47:36 dist-test-slave-n2th krb5kdc[16260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394455, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.194.254@KRBTEST.COM
I20260926 03:47:36.553280 16340 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83b249b98dce4c2c86d27ac0720f4f75: No bootstrap required, opened a new log
I20260926 03:47:36.562407 16343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:36.538501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42876 (local address 127.0.194.254:41965)
0926 03:47:36.539515 (+ 1014us) server_negotiation.cc:206] Beginning negotiation
0926 03:47:36.539535 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0926 03:47:36.539617 (+ 82us) server_negotiation.cc:407] Connection header received
0926 03:47:36.541848 (+ 2231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:47:36.541881 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:47:36.542395 (+ 514us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:47:36.542868 (+ 473us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:47:36.544559 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:47:36.546042 (+ 1483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:47:36.547542 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:47:36.548225 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:47:36.548428 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:47:36.553643 (+ 5215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:47:36.553652 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:47:36.553668 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:47:36.553718 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:47:36.557662 (+ 3944us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:47:36.558728 (+ 1066us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:47:36.558809 (+ 81us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:47:36.558816 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:47:36.559020 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:47:36.559480 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:47:36.559483 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:47:36.559486 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:47:36.560128 (+ 642us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:47:36.560341 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0926 03:47:36.560922 (+ 581us) server_negotiation.cc:299] Negotiation successful
0926 03:47:36.561305 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":734,"thread_start_us":466,"threads_started":1}
I20260926 03:47:36.562947 16340 raft_consensus.cc:348] T 00000000000000000000000000000000 P 83b249b98dce4c2c86d27ac0720f4f75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83b249b98dce4c2c86d27ac0720f4f75" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 41965 } }
I20260926 03:47:36.563356 16340 raft_consensus.cc:374] T 00000000000000000000000000000000 P 83b249b98dce4c2c86d27ac0720f4f75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:47:36.563484 16340 raft_consensus.cc:729] T 00000000000000000000000000000000 P 83b249b98dce4c2c86d27ac0720f4f75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83b249b98dce4c2c86d27ac0720f4f75, State: Initialized, Role: FOLLOWER
I20260926 03:47:36.564078 16340 consensus_queue.cc:260] T 00000000000000000000000000000000 P 83b249b98dce4c2c86d27ac0720f4f75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83b249b98dce4c2c86d27ac0720f4f75" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 41965 } }
I20260926 03:47:36.564391 16340 raft_consensus.cc:388] T 00000000000000000000000000000000 P 83b249b98dce4c2c86d27ac0720f4f75 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:47:36.564546 16340 raft_consensus.cc:482] T 00000000000000000000000000000000 P 83b249b98dce4c2c86d27ac0720f4f75 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:47:36.564757 16340 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 83b249b98dce4c2c86d27ac0720f4f75 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:47:36.567448 16340 raft_consensus.cc:504] T 00000000000000000000000000000000 P 83b249b98dce4c2c86d27ac0720f4f75 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83b249b98dce4c2c86d27ac0720f4f75" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 41965 } }
I20260926 03:47:36.568035 16340 leader_election.cc:302] T 00000000000000000000000000000000 P 83b249b98dce4c2c86d27ac0720f4f75 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 83b249b98dce4c2c86d27ac0720f4f75; no voters:
I20260926 03:47:36.568670 16340 leader_election.cc:288] T 00000000000000000000000000000000 P 83b249b98dce4c2c86d27ac0720f4f75 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:47:36.568806 16345 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 83b249b98dce4c2c86d27ac0720f4f75 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:47:36.569517 16345 raft_consensus.cc:686] T 00000000000000000000000000000000 P 83b249b98dce4c2c86d27ac0720f4f75 [term 1 LEADER]: Becoming Leader. State: Replica: 83b249b98dce4c2c86d27ac0720f4f75, State: Running, Role: LEADER
I20260926 03:47:36.570083 16345 consensus_queue.cc:237] T 00000000000000000000000000000000 P 83b249b98dce4c2c86d27ac0720f4f75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83b249b98dce4c2c86d27ac0720f4f75" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 41965 } }
I20260926 03:47:36.570909 16340 sys_catalog.cc:564] T 00000000000000000000000000000000 P 83b249b98dce4c2c86d27ac0720f4f75 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:47:36.573042 16347 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83b249b98dce4c2c86d27ac0720f4f75 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 83b249b98dce4c2c86d27ac0720f4f75. Latest consensus state: current_term: 1 leader_uuid: "83b249b98dce4c2c86d27ac0720f4f75" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83b249b98dce4c2c86d27ac0720f4f75" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 41965 } } }
I20260926 03:47:36.573544 16347 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83b249b98dce4c2c86d27ac0720f4f75 [sys.catalog]: This master's current role is: LEADER
I20260926 03:47:36.574481 16353 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:47:36.574258 16346 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83b249b98dce4c2c86d27ac0720f4f75 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "83b249b98dce4c2c86d27ac0720f4f75" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83b249b98dce4c2c86d27ac0720f4f75" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 41965 } } }
I20260926 03:47:36.574635 16346 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83b249b98dce4c2c86d27ac0720f4f75 [sys.catalog]: This master's current role is: LEADER
I20260926 03:47:36.580500 16353 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:47:36.590536 16353 catalog_manager.cc:1347] Generated new cluster ID: 8069980440c24e87975da30d412dedd2
I20260926 03:47:36.590674 16353 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:47:36.605298 16353 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:47:36.607375 16353 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:47:36.621727 16353 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 83b249b98dce4c2c86d27ac0720f4f75: Generated new TSK 0
I20260926 03:47:36.622939 16353 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.194.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.194.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.194.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/krb5kdc/kudu_127.0.194.193.keytab.
Entry for principal kudu/127.0.194.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/krb5kdc/kudu_127.0.194.193.keytab.
I20260926 03:47:36.680346 779 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
/tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.194.193:0
--local_ip_for_outbound_sockets=127.0.194.193
--webserver_interface=127.0.194.193
--webserver_port=0
--tserver_master_addrs=127.0.194.254:41965
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/krb5kdc/kudu_127.0.194.193.keytab
--principal=kudu/127.0.194.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.194.212:43861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:47:37.022619 16366 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:47:37.022986 16366 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:47:37.023185 16366 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:47:37.030596 16366 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:47:37.030776 16366 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:47:37.030948 16366 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.194.193
I20260926 03:47:37.041134 16366 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.194.212:43861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/krb5kdc/kudu_127.0.194.193.keytab
--principal=kudu/127.0.194.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.194.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.194.193
--webserver_port=0
--tserver_master_addrs=127.0.194.254:41965
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.194.193
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:47:37.042410 16366 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:47:37.043820 16366 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:47:37.057849 16371 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:47:37.057897 16374 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:47:37.057773 16372 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:47:37.058647 16366 server_base.cc:1034] running on GCE node
I20260926 03:47:37.059376 16366 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:47:37.060377 16366 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:47:37.061609 16366 hybrid_clock.cc:648] HybridClock initialized: now 1790394457061558 us; error 45 us; skew 500 ppm
I20260926 03:47:37.061947 16366 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:47:37 dist-test-slave-n2th krb5kdc[16260](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394457, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:47:37.065376 16366 init.cc:377] Logged in from keytab as kudu/127.0.194.193@KRBTEST.COM (short username kudu)
I20260926 03:47:37.067001 16366 webserver.cc:460] Webserver started at http://127.0.194.193:39049/ using document root <none> and password file <none>
I20260926 03:47:37.067751 16366 fs_manager.cc:359] Metadata directory not provided
I20260926 03:47:37.067906 16366 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:47:37.068236 16366 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:47:37.071410 16366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/minicluster-data/ts-0/data/instance:
uuid: "25395ab024974d1a88518d178c2e2ab2"
format_stamp: "Formatted at 2026-09-26 03:47:37 on dist-test-slave-n2th"
server_key: "512e244a8f11792c6844c3e2977049c0"
server_key_iv: "80df830967f0818d4ef48c077dd0b700"
server_key_version: "encryptionkey@0"
I20260926 03:47:37.072234 16366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/minicluster-data/ts-0/wal/instance:
uuid: "25395ab024974d1a88518d178c2e2ab2"
format_stamp: "Formatted at 2026-09-26 03:47:37 on dist-test-slave-n2th"
server_key: "512e244a8f11792c6844c3e2977049c0"
server_key_iv: "80df830967f0818d4ef48c077dd0b700"
server_key_version: "encryptionkey@0"
I20260926 03:47:37.078266 16366 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260926 03:47:37.082144 16381 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:47:37.083767 16366 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 03:47:37.083977 16366 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/minicluster-data/ts-0/wal
uuid: "25395ab024974d1a88518d178c2e2ab2"
format_stamp: "Formatted at 2026-09-26 03:47:37 on dist-test-slave-n2th"
server_key: "512e244a8f11792c6844c3e2977049c0"
server_key_iv: "80df830967f0818d4ef48c077dd0b700"
server_key_version: "encryptionkey@0"
I20260926 03:47:37.084228 16366 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:47:37.096678 16366 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:47:37.101292 16366 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:47:37.101616 16366 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:47:37.102656 16366 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:47:37.104214 16366 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:47:37.104326 16366 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:47:37.104483 16366 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:47:37.104557 16366 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:47:37.139427 16366 rpc_server.cc:307] RPC server started. Bound to: 127.0.194.193:40383
I20260926 03:47:37.139474 16493 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.194.193:40383 every 8 connection(s)
I20260926 03:47:37.140816 16366 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/minicluster-data/ts-0/data/info.pb
I20260926 03:47:37.144995 779 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu as pid 16366
I20260926 03:47:37.145236 779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/minicluster-data/ts-0/wal/instance
I20260926 03:47:37.146099 779 external_mini_cluster.cc:1428] Setting key 7b040e60a53b5306426ee9c8bd5a63ea
Sep 26 03:47:37 dist-test-slave-n2th krb5kdc[16260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394457, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.193@KRBTEST.COM for kudu/127.0.194.254@KRBTEST.COM
I20260926 03:47:37.164896 16496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:37.143507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.193:34907 (local address 127.0.194.254:41965)
0926 03:47:37.143972 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0926 03:47:37.143977 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:47:37.145162 (+ 1185us) server_negotiation.cc:407] Connection header received
0926 03:47:37.147730 (+ 2568us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:47:37.147738 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:47:37.147814 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:47:37.147918 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:47:37.150069 (+ 2151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:47:37.150912 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:47:37.153467 (+ 2555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:47:37.153870 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:47:37.153969 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:47:37.158894 (+ 4925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:47:37.158900 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:47:37.158903 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:47:37.158940 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:47:37.161566 (+ 2626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:47:37.162419 (+ 853us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:47:37.162424 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:47:37.162427 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:47:37.162551 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:47:37.163056 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:47:37.163063 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:47:37.163067 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:47:37.163350 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:47:37.163514 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0926 03:47:37.164263 (+ 749us) server_negotiation.cc:299] Negotiation successful
0926 03:47:37.164518 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":265,"threads_started":1}
I20260926 03:47:37.165931 16497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:37.144013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:41965 (local address 127.0.194.193:34907)
0926 03:47:37.144860 (+ 847us) negotiation.cc:107] Waiting for socket to connect
0926 03:47:37.144923 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0926 03:47:37.147336 (+ 2413us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:47:37.148205 (+ 869us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:47:37.148214 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:47:37.148978 (+ 764us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:47:37.149870 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:47:37.149900 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:47:37.151083 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:47:37.151091 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:47:37.153301 (+ 2210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:47:37.153326 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:47:37.154123 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:47:37.154132 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:47:37.154484 (+ 352us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:47:37.155305 (+ 821us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:47:37.155341 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:47:37.158589 (+ 3248us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:47:37.161755 (+ 3166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:47:37.161765 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:47:37.161784 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:47:37.162261 (+ 477us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:47:37.162668 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:47:37.162673 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:47:37.162677 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:47:37.162943 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:47:37.163482 (+ 539us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:47:37.163490 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:47:37.163881 (+ 391us) client_negotiation.cc:769] Sending connection context
0926 03:47:37.164158 (+ 277us) client_negotiation.cc:240] Negotiation successful
0926 03:47:37.164756 (+ 598us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":274,"threads_started":1}
I20260926 03:47:37.167867 16494 heartbeater.cc:347] Connected to a master server at 127.0.194.254:41965
I20260926 03:47:37.168224 16494 heartbeater.cc:464] Registering TS with master...
I20260926 03:47:37.168989 16494 heartbeater.cc:511] Master 127.0.194.254:41965 requested a full tablet report, sending...
I20260926 03:47:37.171440 16305 ts_manager.cc:194] Registered new tserver with Master: 25395ab024974d1a88518d178c2e2ab2 (127.0.194.193:40383)
WARNING: no policy specified for kudu/127.0.194.194@KRBTEST.COM; defaulting to no policy
I20260926 03:47:37.173192 16305 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.194.193@KRBTEST.COM'} at 127.0.194.193:34907
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.194.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.194.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/krb5kdc/kudu_127.0.194.194.keytab.
Entry for principal kudu/127.0.194.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/krb5kdc/kudu_127.0.194.194.keytab.
I20260926 03:47:37.202436 779 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
/tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.194.194:0
--local_ip_for_outbound_sockets=127.0.194.194
--webserver_interface=127.0.194.194
--webserver_port=0
--tserver_master_addrs=127.0.194.254:41965
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/krb5kdc/kudu_127.0.194.194.keytab
--principal=kudu/127.0.194.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.194.212:43861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:47:37.535079 16500 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:47:37.535408 16500 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:47:37.535584 16500 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:47:37.543828 16500 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:47:37.543980 16500 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:47:37.544220 16500 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.194.194
I20260926 03:47:37.555119 16500 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.194.212:43861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/krb5kdc/kudu_127.0.194.194.keytab
--principal=kudu/127.0.194.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.194.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.194.194
--webserver_port=0
--tserver_master_addrs=127.0.194.254:41965
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.194.194
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:47:37.556388 16500 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:47:37.557639 16500 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:47:37.570926 16505 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:47:37.570950 16508 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:47:37.570869 16506 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:47:37.571913 16500 server_base.cc:1034] running on GCE node
I20260926 03:47:37.572621 16500 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:47:37.573607 16500 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:47:37.574843 16500 hybrid_clock.cc:648] HybridClock initialized: now 1790394457574764 us; error 70 us; skew 500 ppm
I20260926 03:47:37.575174 16500 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:47:37 dist-test-slave-n2th krb5kdc[16260](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394457, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:47:37.578521 16500 init.cc:377] Logged in from keytab as kudu/127.0.194.194@KRBTEST.COM (short username kudu)
I20260926 03:47:37.580044 16500 webserver.cc:460] Webserver started at http://127.0.194.194:37899/ using document root <none> and password file <none>
I20260926 03:47:37.580744 16500 fs_manager.cc:359] Metadata directory not provided
I20260926 03:47:37.580875 16500 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:47:37.581183 16500 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:47:37.584395 16500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/minicluster-data/ts-1/data/instance:
uuid: "b5327dcf8cbc4adb804842b27d7ab913"
format_stamp: "Formatted at 2026-09-26 03:47:37 on dist-test-slave-n2th"
server_key: "6001f474c3d8de06bca8de24bdb8b31d"
server_key_iv: "308b140ceb06480085467a7e42793253"
server_key_version: "encryptionkey@0"
I20260926 03:47:37.585237 16500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/minicluster-data/ts-1/wal/instance:
uuid: "b5327dcf8cbc4adb804842b27d7ab913"
format_stamp: "Formatted at 2026-09-26 03:47:37 on dist-test-slave-n2th"
server_key: "6001f474c3d8de06bca8de24bdb8b31d"
server_key_iv: "308b140ceb06480085467a7e42793253"
server_key_version: "encryptionkey@0"
I20260926 03:47:37.590924 16500 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260926 03:47:37.594558 16515 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:47:37.595988 16500 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 03:47:37.596180 16500 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/minicluster-data/ts-1/wal
uuid: "b5327dcf8cbc4adb804842b27d7ab913"
format_stamp: "Formatted at 2026-09-26 03:47:37 on dist-test-slave-n2th"
server_key: "6001f474c3d8de06bca8de24bdb8b31d"
server_key_iv: "308b140ceb06480085467a7e42793253"
server_key_version: "encryptionkey@0"
I20260926 03:47:37.596400 16500 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:47:37.612874 16500 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:47:37.617486 16500 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:47:37.617806 16500 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:47:37.618877 16500 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:47:37.620280 16500 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:47:37.620388 16500 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:47:37.620545 16500 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:47:37.620637 16500 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:47:37.654544 16500 rpc_server.cc:307] RPC server started. Bound to: 127.0.194.194:41349
I20260926 03:47:37.654587 16627 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.194.194:41349 every 8 connection(s)
I20260926 03:47:37.655956 16500 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/minicluster-data/ts-1/data/info.pb
I20260926 03:47:37.656438 779 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu as pid 16500
I20260926 03:47:37.656651 779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/minicluster-data/ts-1/wal/instance
I20260926 03:47:37.657234 779 external_mini_cluster.cc:1428] Setting key 4a2bde5ee9f2f42c9682f40e97929937
Sep 26 03:47:37 dist-test-slave-n2th krb5kdc[16260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394457, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.194@KRBTEST.COM for kudu/127.0.194.254@KRBTEST.COM
I20260926 03:47:37.679039 16496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:37.658870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.194:56955 (local address 127.0.194.254:41965)
0926 03:47:37.659041 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0926 03:47:37.659046 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:47:37.660547 (+ 1501us) server_negotiation.cc:407] Connection header received
0926 03:47:37.662436 (+ 1889us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:47:37.662442 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:47:37.662537 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:47:37.662649 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:47:37.664286 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:47:37.665015 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:47:37.666816 (+ 1801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:47:37.667660 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:47:37.667821 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:47:37.671582 (+ 3761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:47:37.671591 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:47:37.671600 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:47:37.671656 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:47:37.675387 (+ 3731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:47:37.676225 (+ 838us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:47:37.676233 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:47:37.676239 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:47:37.676416 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:47:37.676865 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:47:37.676873 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:47:37.676879 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:47:37.677290 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:47:37.677452 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0926 03:47:37.678239 (+ 787us) server_negotiation.cc:299] Negotiation successful
0926 03:47:37.678528 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 03:47:37.679875 16630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:37.659581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:41965 (local address 127.0.194.194:56955)
0926 03:47:37.660395 (+ 814us) negotiation.cc:107] Waiting for socket to connect
0926 03:47:37.660434 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 03:47:37.662148 (+ 1714us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:47:37.662957 (+ 809us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:47:37.662966 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:47:37.663505 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:47:37.664107 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:47:37.664125 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:47:37.665187 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:47:37.665191 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:47:37.666589 (+ 1398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:47:37.666609 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:47:37.667893 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:47:37.667898 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:47:37.668105 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:47:37.668839 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:47:37.668878 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:47:37.671334 (+ 2456us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:47:37.675540 (+ 4206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:47:37.675548 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:47:37.675562 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:47:37.676087 (+ 525us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:47:37.676538 (+ 451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:47:37.676541 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:47:37.676544 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:47:37.676737 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:47:37.677422 (+ 685us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:47:37.677428 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:47:37.677832 (+ 404us) client_negotiation.cc:769] Sending connection context
0926 03:47:37.678079 (+ 247us) client_negotiation.cc:240] Negotiation successful
0926 03:47:37.678500 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":564,"thread_start_us":332,"threads_started":1}
I20260926 03:47:37.681509 16628 heartbeater.cc:347] Connected to a master server at 127.0.194.254:41965
I20260926 03:47:37.681883 16628 heartbeater.cc:464] Registering TS with master...
I20260926 03:47:37.682924 16628 heartbeater.cc:511] Master 127.0.194.254:41965 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.194.195@KRBTEST.COM; defaulting to no policy
I20260926 03:47:37.684664 16305 ts_manager.cc:194] Registered new tserver with Master: b5327dcf8cbc4adb804842b27d7ab913 (127.0.194.194:41349)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.194.195@KRBTEST.COM" created.
I20260926 03:47:37.685701 16305 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.194.194@KRBTEST.COM'} at 127.0.194.194:56955
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.194.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/krb5kdc/kudu_127.0.194.195.keytab.
Entry for principal kudu/127.0.194.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/krb5kdc/kudu_127.0.194.195.keytab.
I20260926 03:47:37.714881 779 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
/tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.194.195:0
--local_ip_for_outbound_sockets=127.0.194.195
--webserver_interface=127.0.194.195
--webserver_port=0
--tserver_master_addrs=127.0.194.254:41965
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/krb5kdc/kudu_127.0.194.195.keytab
--principal=kudu/127.0.194.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.194.212:43861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:47:38.053612 16633 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:47:38.053946 16633 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:47:38.054112 16633 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:47:38.061581 16633 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:47:38.061786 16633 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:47:38.061969 16633 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.194.195
I20260926 03:47:38.072351 16633 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.194.212:43861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/krb5kdc/kudu_127.0.194.195.keytab
--principal=kudu/127.0.194.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.194.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.194.195
--webserver_port=0
--tserver_master_addrs=127.0.194.254:41965
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.194.195
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:47:38.073779 16633 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:47:38.075197 16633 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:47:38.088768 16639 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:47:38.088891 16641 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:47:38.088791 16638 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:47:38.090086 16633 server_base.cc:1034] running on GCE node
I20260926 03:47:38.090809 16633 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:47:38.091892 16633 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:47:38.093170 16633 hybrid_clock.cc:648] HybridClock initialized: now 1790394458093104 us; error 62 us; skew 500 ppm
I20260926 03:47:38.093544 16633 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:47:38 dist-test-slave-n2th krb5kdc[16260](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394458, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:47:38.096851 16633 init.cc:377] Logged in from keytab as kudu/127.0.194.195@KRBTEST.COM (short username kudu)
I20260926 03:47:38.098419 16633 webserver.cc:460] Webserver started at http://127.0.194.195:35835/ using document root <none> and password file <none>
I20260926 03:47:38.099208 16633 fs_manager.cc:359] Metadata directory not provided
I20260926 03:47:38.099370 16633 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:47:38.099771 16633 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:47:38.103084 16633 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/minicluster-data/ts-2/data/instance:
uuid: "e4ed0f05ec9949a5a78562546d5566d2"
format_stamp: "Formatted at 2026-09-26 03:47:38 on dist-test-slave-n2th"
server_key: "2e953a21f430b59eee8dcee318951152"
server_key_iv: "5324ade121b9e7de9d7c29bf0d51e50e"
server_key_version: "encryptionkey@0"
I20260926 03:47:38.103989 16633 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/minicluster-data/ts-2/wal/instance:
uuid: "e4ed0f05ec9949a5a78562546d5566d2"
format_stamp: "Formatted at 2026-09-26 03:47:38 on dist-test-slave-n2th"
server_key: "2e953a21f430b59eee8dcee318951152"
server_key_iv: "5324ade121b9e7de9d7c29bf0d51e50e"
server_key_version: "encryptionkey@0"
I20260926 03:47:38.109628 16633 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 03:47:38.113361 16648 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:47:38.114836 16633 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 03:47:38.115034 16633 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/minicluster-data/ts-2/wal
uuid: "e4ed0f05ec9949a5a78562546d5566d2"
format_stamp: "Formatted at 2026-09-26 03:47:38 on dist-test-slave-n2th"
server_key: "2e953a21f430b59eee8dcee318951152"
server_key_iv: "5324ade121b9e7de9d7c29bf0d51e50e"
server_key_version: "encryptionkey@0"
I20260926 03:47:38.115250 16633 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:47:38.140112 16633 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:47:38.144881 16633 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:47:38.145195 16633 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:47:38.146198 16633 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:47:38.147646 16633 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:47:38.147753 16633 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:47:38.147940 16633 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:47:38.148026 16633 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:47:38.176576 16494 heartbeater.cc:503] Master 127.0.194.254:41965 was elected leader, sending a full tablet report...
I20260926 03:47:38.182583 16633 rpc_server.cc:307] RPC server started. Bound to: 127.0.194.195:41479
I20260926 03:47:38.182641 16760 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.194.195:41479 every 8 connection(s)
I20260926 03:47:38.184008 16633 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/minicluster-data/ts-2/data/info.pb
I20260926 03:47:38.185400 779 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu as pid 16633
I20260926 03:47:38.185595 779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/minicluster-data/ts-2/wal/instance
I20260926 03:47:38.186173 779 external_mini_cluster.cc:1428] Setting key 04bf100bde1a9fb4c4a7e4c932bf3b78
Sep 26 03:47:38 dist-test-slave-n2th krb5kdc[16260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394458, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.195@KRBTEST.COM for kudu/127.0.194.254@KRBTEST.COM
I20260926 03:47:38.206082 16763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:38.186993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.195:35867 (local address 127.0.194.254:41965)
0926 03:47:38.187418 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0926 03:47:38.187423 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:47:38.188949 (+ 1526us) server_negotiation.cc:407] Connection header received
0926 03:47:38.191192 (+ 2243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:47:38.191196 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:47:38.191265 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:47:38.191362 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:47:38.192912 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:47:38.193701 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:47:38.195321 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:47:38.195713 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:47:38.195806 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:47:38.199308 (+ 3502us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:47:38.199313 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:47:38.199316 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:47:38.199354 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:47:38.202981 (+ 3627us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:47:38.203837 (+ 856us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:47:38.203842 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:47:38.203844 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:47:38.203992 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:47:38.204452 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:47:38.204455 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:47:38.204457 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:47:38.204753 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:47:38.204881 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0926 03:47:38.205625 (+ 744us) server_negotiation.cc:299] Negotiation successful
0926 03:47:38.205808 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":217,"threads_started":1}
I20260926 03:47:38.207190 16764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:38.187779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:41965 (local address 127.0.194.195:35867)
0926 03:47:38.188774 (+ 995us) negotiation.cc:107] Waiting for socket to connect
0926 03:47:38.188853 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0926 03:47:38.190943 (+ 2090us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:47:38.191565 (+ 622us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:47:38.191573 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:47:38.192185 (+ 612us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:47:38.192762 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:47:38.192780 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:47:38.193882 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:47:38.193887 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:47:38.195201 (+ 1314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:47:38.195213 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:47:38.195914 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:47:38.195919 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:47:38.196112 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:47:38.196757 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:47:38.196794 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:47:38.199067 (+ 2273us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:47:38.203151 (+ 4084us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:47:38.203158 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:47:38.203175 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:47:38.203684 (+ 509us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:47:38.204108 (+ 424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:47:38.204111 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:47:38.204113 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:47:38.204323 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:47:38.204870 (+ 547us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:47:38.204876 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:47:38.205236 (+ 360us) client_negotiation.cc:769] Sending connection context
0926 03:47:38.205490 (+ 254us) client_negotiation.cc:240] Negotiation successful
0926 03:47:38.205910 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":723,"thread_start_us":461,"threads_started":1}
I20260926 03:47:38.208671 16761 heartbeater.cc:347] Connected to a master server at 127.0.194.254:41965
I20260926 03:47:38.208976 16761 heartbeater.cc:464] Registering TS with master...
I20260926 03:47:38.209671 16761 heartbeater.cc:511] Master 127.0.194.254:41965 requested a full tablet report, sending...
I20260926 03:47:38.211172 16305 ts_manager.cc:194] Registered new tserver with Master: e4ed0f05ec9949a5a78562546d5566d2 (127.0.194.195:41479)
I20260926 03:47:38.212005 16305 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.194.195@KRBTEST.COM'} at 127.0.194.195:35867
I20260926 03:47:38.215413 779 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 03:47:38.233022 16763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:38.220260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52152 (local address 127.0.194.254:41965)
0926 03:47:38.220424 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0926 03:47:38.220428 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:47:38.220955 (+ 527us) server_negotiation.cc:407] Connection header received
0926 03:47:38.221123 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:47:38.221127 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:47:38.221193 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:47:38.221304 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:47:38.222698 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:47:38.223492 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:47:38.224927 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:47:38.225353 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:47:38.225450 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:47:38.227982 (+ 2532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:47:38.227987 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:47:38.227990 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:47:38.228025 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:47:38.230628 (+ 2603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:47:38.231440 (+ 812us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:47:38.231444 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:47:38.231446 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:47:38.231563 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:47:38.231881 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:47:38.231884 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:47:38.231886 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:47:38.232202 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:47:38.232395 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0926 03:47:38.232616 (+ 221us) server_negotiation.cc:299] Negotiation successful
0926 03:47:38.232790 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 03:47:38.237329 16305 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52152:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 03:47:38.244490 16305 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 03:47:38.269868 16774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:38.257607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.194:41349 (local address 127.0.0.1:48454)
0926 03:47:38.258616 (+ 1009us) negotiation.cc:107] Waiting for socket to connect
0926 03:47:38.258674 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0926 03:47:38.258909 (+ 235us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:47:38.262662 (+ 3753us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:47:38.262669 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:47:38.262696 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:47:38.263366 (+ 670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:47:38.263381 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:47:38.265002 (+ 1621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:47:38.265007 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:47:38.266945 (+ 1938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:47:38.266957 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:47:38.268733 (+ 1776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:47:38.268740 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:47:38.268972 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:47:38.269027 (+ 55us) client_negotiation.cc:769] Sending connection context
0926 03:47:38.269145 (+ 118us) client_negotiation.cc:240] Negotiation successful
0926 03:47:38.269356 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":722,"mutex_wait_us":19,"spinlock_wait_cycles":1792,"thread_start_us":505,"threads_started":1}
I20260926 03:47:38.270396 16778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:38.259228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.195:41479 (local address 127.0.0.1:43140)
0926 03:47:38.261425 (+ 2197us) negotiation.cc:107] Waiting for socket to connect
0926 03:47:38.261446 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 03:47:38.261577 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:47:38.262577 (+ 1000us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:47:38.262581 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:47:38.262628 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:47:38.263378 (+ 750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:47:38.263387 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:47:38.265359 (+ 1972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:47:38.265365 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:47:38.267698 (+ 2333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:47:38.267709 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:47:38.269457 (+ 1748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:47:38.269463 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:47:38.269650 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:47:38.269725 (+ 75us) client_negotiation.cc:769] Sending connection context
0926 03:47:38.269832 (+ 107us) client_negotiation.cc:240] Negotiation successful
0926 03:47:38.269967 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2056,"mutex_wait_us":49,"thread_start_us":1909,"threads_started":1}
I20260926 03:47:38.270576 16775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:38.257890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48454 (local address 127.0.194.194:41349)
0926 03:47:38.260538 (+ 2648us) server_negotiation.cc:206] Beginning negotiation
0926 03:47:38.260552 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 03:47:38.260590 (+ 38us) server_negotiation.cc:407] Connection header received
0926 03:47:38.260727 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:47:38.260740 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:47:38.262403 (+ 1663us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:47:38.262664 (+ 261us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:47:38.263611 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:47:38.264817 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:47:38.267098 (+ 2281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:47:38.268490 (+ 1392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:47:38.269740 (+ 1250us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:47:38.269931 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0926 03:47:38.270025 (+ 94us) server_negotiation.cc:299] Negotiation successful
0926 03:47:38.270203 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2440,"thread_start_us":2234,"threads_started":1}
I20260926 03:47:38.270963 16777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:38.259139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39352 (local address 127.0.194.193:40383)
0926 03:47:38.261768 (+ 2629us) server_negotiation.cc:206] Beginning negotiation
0926 03:47:38.261787 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 03:47:38.261811 (+ 24us) server_negotiation.cc:407] Connection header received
0926 03:47:38.261887 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:47:38.261893 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:47:38.262090 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:47:38.262294 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:47:38.263073 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:47:38.264614 (+ 1541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:47:38.267650 (+ 3036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:47:38.269130 (+ 1480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:47:38.269255 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:47:38.269503 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0926 03:47:38.270420 (+ 917us) server_negotiation.cc:299] Negotiation successful
0926 03:47:38.270599 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2458,"thread_start_us":2331,"threads_started":1}
I20260926 03:47:38.271225 16779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:38.260017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43140 (local address 127.0.194.195:41479)
0926 03:47:38.260932 (+ 915us) server_negotiation.cc:206] Beginning negotiation
0926 03:47:38.260944 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 03:47:38.261715 (+ 771us) server_negotiation.cc:407] Connection header received
0926 03:47:38.261823 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:47:38.261832 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:47:38.262091 (+ 259us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:47:38.262296 (+ 205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:47:38.263599 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:47:38.265195 (+ 1596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:47:38.267844 (+ 2649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:47:38.269257 (+ 1413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:47:38.269407 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:47:38.269638 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0926 03:47:38.270690 (+ 1052us) server_negotiation.cc:299] Negotiation successful
0926 03:47:38.270908 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":697,"thread_start_us":531,"threads_started":1}
I20260926 03:47:38.271474 16776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:38.258456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.193:40383 (local address 127.0.0.1:39352)
0926 03:47:38.259727 (+ 1271us) negotiation.cc:107] Waiting for socket to connect
0926 03:47:38.259757 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0926 03:47:38.259899 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:47:38.262305 (+ 2406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:47:38.262312 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:47:38.262350 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:47:38.262885 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:47:38.262911 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:47:38.264774 (+ 1863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:47:38.264780 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:47:38.267432 (+ 2652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:47:38.267447 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:47:38.269647 (+ 2200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:47:38.269652 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:47:38.269863 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:47:38.269904 (+ 41us) client_negotiation.cc:769] Sending connection context
0926 03:47:38.270290 (+ 386us) client_negotiation.cc:240] Negotiation successful
0926 03:47:38.270442 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1119,"mutex_wait_us":1,"thread_start_us":978,"threads_started":1}
I20260926 03:47:38.274269 16563 tablet_service.cc:1467] Processing CreateTablet for tablet 36771bd187224ce4b31131e61a01e426 (DEFAULT_TABLE table=test-table [id=47c3a0bd5c1d4d63b606738006ef45ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:47:38.274832 16429 tablet_service.cc:1467] Processing CreateTablet for tablet 36771bd187224ce4b31131e61a01e426 (DEFAULT_TABLE table=test-table [id=47c3a0bd5c1d4d63b606738006ef45ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:47:38.274823 16696 tablet_service.cc:1467] Processing CreateTablet for tablet 36771bd187224ce4b31131e61a01e426 (DEFAULT_TABLE table=test-table [id=47c3a0bd5c1d4d63b606738006ef45ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:47:38.275738 16563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36771bd187224ce4b31131e61a01e426. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:47:38.276218 16429 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36771bd187224ce4b31131e61a01e426. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:47:38.276217 16696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36771bd187224ce4b31131e61a01e426. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:47:38.288822 16780 tablet_bootstrap.cc:492] T 36771bd187224ce4b31131e61a01e426 P b5327dcf8cbc4adb804842b27d7ab913: Bootstrap starting.
I20260926 03:47:38.291239 16781 tablet_bootstrap.cc:492] T 36771bd187224ce4b31131e61a01e426 P 25395ab024974d1a88518d178c2e2ab2: Bootstrap starting.
I20260926 03:47:38.292528 16780 tablet_bootstrap.cc:654] T 36771bd187224ce4b31131e61a01e426 P b5327dcf8cbc4adb804842b27d7ab913: Neither blocks nor log segments found. Creating new log.
I20260926 03:47:38.293871 16780 log.cc:824] T 36771bd187224ce4b31131e61a01e426 P b5327dcf8cbc4adb804842b27d7ab913: Log is configured to *not* fsync() on all Append() calls
I20260926 03:47:38.294947 16781 tablet_bootstrap.cc:654] T 36771bd187224ce4b31131e61a01e426 P 25395ab024974d1a88518d178c2e2ab2: Neither blocks nor log segments found. Creating new log.
I20260926 03:47:38.296531 16781 log.cc:824] T 36771bd187224ce4b31131e61a01e426 P 25395ab024974d1a88518d178c2e2ab2: Log is configured to *not* fsync() on all Append() calls
I20260926 03:47:38.298422 16782 tablet_bootstrap.cc:492] T 36771bd187224ce4b31131e61a01e426 P e4ed0f05ec9949a5a78562546d5566d2: Bootstrap starting.
I20260926 03:47:38.300629 16781 tablet_bootstrap.cc:492] T 36771bd187224ce4b31131e61a01e426 P 25395ab024974d1a88518d178c2e2ab2: No bootstrap required, opened a new log
I20260926 03:47:38.301363 16781 ts_tablet_manager.cc:1397] T 36771bd187224ce4b31131e61a01e426 P 25395ab024974d1a88518d178c2e2ab2: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.003s
I20260926 03:47:38.302485 16782 tablet_bootstrap.cc:654] T 36771bd187224ce4b31131e61a01e426 P e4ed0f05ec9949a5a78562546d5566d2: Neither blocks nor log segments found. Creating new log.
I20260926 03:47:38.302999 16780 tablet_bootstrap.cc:492] T 36771bd187224ce4b31131e61a01e426 P b5327dcf8cbc4adb804842b27d7ab913: No bootstrap required, opened a new log
I20260926 03:47:38.303294 16780 ts_tablet_manager.cc:1397] T 36771bd187224ce4b31131e61a01e426 P b5327dcf8cbc4adb804842b27d7ab913: Time spent bootstrapping tablet: real 0.015s user 0.009s sys 0.004s
I20260926 03:47:38.303756 16782 log.cc:824] T 36771bd187224ce4b31131e61a01e426 P e4ed0f05ec9949a5a78562546d5566d2: Log is configured to *not* fsync() on all Append() calls
I20260926 03:47:38.312564 16781 raft_consensus.cc:348] T 36771bd187224ce4b31131e61a01e426 P 25395ab024974d1a88518d178c2e2ab2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5327dcf8cbc4adb804842b27d7ab913" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 41349 } } peers { permanent_uuid: "e4ed0f05ec9949a5a78562546d5566d2" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 41479 } } peers { permanent_uuid: "25395ab024974d1a88518d178c2e2ab2" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 40383 } }
I20260926 03:47:38.313098 16781 raft_consensus.cc:374] T 36771bd187224ce4b31131e61a01e426 P 25395ab024974d1a88518d178c2e2ab2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:47:38.313198 16781 raft_consensus.cc:729] T 36771bd187224ce4b31131e61a01e426 P 25395ab024974d1a88518d178c2e2ab2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25395ab024974d1a88518d178c2e2ab2, State: Initialized, Role: FOLLOWER
I20260926 03:47:38.313799 16782 tablet_bootstrap.cc:492] T 36771bd187224ce4b31131e61a01e426 P e4ed0f05ec9949a5a78562546d5566d2: No bootstrap required, opened a new log
I20260926 03:47:38.313884 16781 consensus_queue.cc:260] T 36771bd187224ce4b31131e61a01e426 P 25395ab024974d1a88518d178c2e2ab2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5327dcf8cbc4adb804842b27d7ab913" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 41349 } } peers { permanent_uuid: "e4ed0f05ec9949a5a78562546d5566d2" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 41479 } } peers { permanent_uuid: "25395ab024974d1a88518d178c2e2ab2" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 40383 } }
I20260926 03:47:38.314106 16782 ts_tablet_manager.cc:1397] T 36771bd187224ce4b31131e61a01e426 P e4ed0f05ec9949a5a78562546d5566d2: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.010s
I20260926 03:47:38.314033 16780 raft_consensus.cc:348] T 36771bd187224ce4b31131e61a01e426 P b5327dcf8cbc4adb804842b27d7ab913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5327dcf8cbc4adb804842b27d7ab913" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 41349 } } peers { permanent_uuid: "e4ed0f05ec9949a5a78562546d5566d2" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 41479 } } peers { permanent_uuid: "25395ab024974d1a88518d178c2e2ab2" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 40383 } }
I20260926 03:47:38.314554 16780 raft_consensus.cc:374] T 36771bd187224ce4b31131e61a01e426 P b5327dcf8cbc4adb804842b27d7ab913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:47:38.314697 16780 raft_consensus.cc:729] T 36771bd187224ce4b31131e61a01e426 P b5327dcf8cbc4adb804842b27d7ab913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5327dcf8cbc4adb804842b27d7ab913, State: Initialized, Role: FOLLOWER
I20260926 03:47:38.315668 16780 consensus_queue.cc:260] T 36771bd187224ce4b31131e61a01e426 P b5327dcf8cbc4adb804842b27d7ab913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5327dcf8cbc4adb804842b27d7ab913" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 41349 } } peers { permanent_uuid: "e4ed0f05ec9949a5a78562546d5566d2" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 41479 } } peers { permanent_uuid: "25395ab024974d1a88518d178c2e2ab2" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 40383 } }
I20260926 03:47:38.317600 16628 heartbeater.cc:503] Master 127.0.194.254:41965 was elected leader, sending a full tablet report...
I20260926 03:47:38.319533 16780 ts_tablet_manager.cc:1428] T 36771bd187224ce4b31131e61a01e426 P b5327dcf8cbc4adb804842b27d7ab913: Time spent starting tablet: real 0.016s user 0.015s sys 0.000s
I20260926 03:47:38.321429 16781 ts_tablet_manager.cc:1428] T 36771bd187224ce4b31131e61a01e426 P 25395ab024974d1a88518d178c2e2ab2: Time spent starting tablet: real 0.020s user 0.013s sys 0.003s
I20260926 03:47:38.324690 16782 raft_consensus.cc:348] T 36771bd187224ce4b31131e61a01e426 P e4ed0f05ec9949a5a78562546d5566d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5327dcf8cbc4adb804842b27d7ab913" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 41349 } } peers { permanent_uuid: "e4ed0f05ec9949a5a78562546d5566d2" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 41479 } } peers { permanent_uuid: "25395ab024974d1a88518d178c2e2ab2" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 40383 } }
I20260926 03:47:38.325114 16782 raft_consensus.cc:374] T 36771bd187224ce4b31131e61a01e426 P e4ed0f05ec9949a5a78562546d5566d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:47:38.325238 16782 raft_consensus.cc:729] T 36771bd187224ce4b31131e61a01e426 P e4ed0f05ec9949a5a78562546d5566d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4ed0f05ec9949a5a78562546d5566d2, State: Initialized, Role: FOLLOWER
I20260926 03:47:38.325912 16782 consensus_queue.cc:260] T 36771bd187224ce4b31131e61a01e426 P e4ed0f05ec9949a5a78562546d5566d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5327dcf8cbc4adb804842b27d7ab913" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 41349 } } peers { permanent_uuid: "e4ed0f05ec9949a5a78562546d5566d2" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 41479 } } peers { permanent_uuid: "25395ab024974d1a88518d178c2e2ab2" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 40383 } }
I20260926 03:47:38.327036 16761 heartbeater.cc:503] Master 127.0.194.254:41965 was elected leader, sending a full tablet report...
I20260926 03:47:38.327800 16782 ts_tablet_manager.cc:1428] T 36771bd187224ce4b31131e61a01e426 P e4ed0f05ec9949a5a78562546d5566d2: Time spent starting tablet: real 0.014s user 0.009s sys 0.006s
W20260926 03:47:38.393282 16495 tablet.cc:2369] T 36771bd187224ce4b31131e61a01e426 P 25395ab024974d1a88518d178c2e2ab2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 03:47:38.408313 16629 tablet.cc:2369] T 36771bd187224ce4b31131e61a01e426 P b5327dcf8cbc4adb804842b27d7ab913: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 03:47:38.435793 16762 tablet.cc:2369] T 36771bd187224ce4b31131e61a01e426 P e4ed0f05ec9949a5a78562546d5566d2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 03:47:38.553889 16788 raft_consensus.cc:482] T 36771bd187224ce4b31131e61a01e426 P e4ed0f05ec9949a5a78562546d5566d2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 03:47:38.554251 16788 raft_consensus.cc:504] T 36771bd187224ce4b31131e61a01e426 P e4ed0f05ec9949a5a78562546d5566d2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5327dcf8cbc4adb804842b27d7ab913" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 41349 } } peers { permanent_uuid: "e4ed0f05ec9949a5a78562546d5566d2" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 41479 } } peers { permanent_uuid: "25395ab024974d1a88518d178c2e2ab2" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 40383 } }
I20260926 03:47:38.556159 16788 leader_election.cc:288] T 36771bd187224ce4b31131e61a01e426 P e4ed0f05ec9949a5a78562546d5566d2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b5327dcf8cbc4adb804842b27d7ab913 (127.0.194.194:41349), 25395ab024974d1a88518d178c2e2ab2 (127.0.194.193:40383)
I20260926 03:47:38.563827 16777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:38.556665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.195:58675 (local address 127.0.194.193:40383)
0926 03:47:38.556913 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0926 03:47:38.556923 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 03:47:38.556953 (+ 30us) server_negotiation.cc:407] Connection header received
0926 03:47:38.557175 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:47:38.557180 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:47:38.557277 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:47:38.557375 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:47:38.557833 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:47:38.558846 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:47:38.561485 (+ 2639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:47:38.562722 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:47:38.562950 (+ 228us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:47:38.563032 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 03:47:38.563336 (+ 304us) server_negotiation.cc:299] Negotiation successful
0926 03:47:38.563451 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 03:47:38.563817 16764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:38.556509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.193:40383 (local address 127.0.194.195:58675)
0926 03:47:38.556845 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0926 03:47:38.556866 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 03:47:38.557005 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:47:38.557428 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:47:38.557435 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:47:38.557476 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:47:38.557690 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:47:38.557701 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:47:38.559054 (+ 1353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:47:38.559059 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:47:38.561304 (+ 2245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:47:38.561320 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:47:38.562940 (+ 1620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:47:38.562945 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:47:38.563111 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:47:38.563153 (+ 42us) client_negotiation.cc:769] Sending connection context
0926 03:47:38.563289 (+ 136us) client_negotiation.cc:240] Negotiation successful
0926 03:47:38.563453 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"mutex_wait_us":29}
I20260926 03:47:38.564198 16775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:38.556556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.195:59369 (local address 127.0.194.194:41349)
0926 03:47:38.556883 (+ 327us) server_negotiation.cc:206] Beginning negotiation
0926 03:47:38.556891 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:47:38.557532 (+ 641us) server_negotiation.cc:407] Connection header received
0926 03:47:38.557671 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:47:38.557674 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:47:38.557730 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:47:38.557869 (+ 139us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:47:38.558345 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:47:38.559313 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:47:38.561726 (+ 2413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:47:38.563024 (+ 1298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:47:38.563165 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:47:38.563216 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0926 03:47:38.563736 (+ 520us) server_negotiation.cc:299] Negotiation successful
0926 03:47:38.563916 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":190}
I20260926 03:47:38.564260 16789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:38.556562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.194:41349 (local address 127.0.194.195:59369)
0926 03:47:38.557411 (+ 849us) negotiation.cc:107] Waiting for socket to connect
0926 03:47:38.557432 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 03:47:38.557556 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:47:38.557893 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:47:38.557895 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:47:38.557915 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:47:38.558183 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:47:38.558190 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:47:38.559474 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:47:38.559479 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:47:38.561565 (+ 2086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:47:38.561575 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:47:38.563280 (+ 1705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:47:38.563287 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:47:38.563459 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:47:38.563489 (+ 30us) client_negotiation.cc:769] Sending connection context
0926 03:47:38.563582 (+ 93us) client_negotiation.cc:240] Negotiation successful
0926 03:47:38.563717 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":721,"mutex_wait_us":37,"thread_start_us":499,"threads_started":1}
I20260926 03:47:38.565423 16583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36771bd187224ce4b31131e61a01e426" candidate_uuid: "e4ed0f05ec9949a5a78562546d5566d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5327dcf8cbc4adb804842b27d7ab913" is_pre_election: true
I20260926 03:47:38.565423 16449 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36771bd187224ce4b31131e61a01e426" candidate_uuid: "e4ed0f05ec9949a5a78562546d5566d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25395ab024974d1a88518d178c2e2ab2" is_pre_election: true
I20260926 03:47:38.566061 16583 raft_consensus.cc:2454] T 36771bd187224ce4b31131e61a01e426 P b5327dcf8cbc4adb804842b27d7ab913 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4ed0f05ec9949a5a78562546d5566d2 in term 0.
I20260926 03:47:38.566061 16449 raft_consensus.cc:2454] T 36771bd187224ce4b31131e61a01e426 P 25395ab024974d1a88518d178c2e2ab2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4ed0f05ec9949a5a78562546d5566d2 in term 0.
I20260926 03:47:38.566990 16650 leader_election.cc:302] T 36771bd187224ce4b31131e61a01e426 P e4ed0f05ec9949a5a78562546d5566d2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 25395ab024974d1a88518d178c2e2ab2, b5327dcf8cbc4adb804842b27d7ab913, e4ed0f05ec9949a5a78562546d5566d2; no voters:
I20260926 03:47:38.567530 16788 raft_consensus.cc:2781] T 36771bd187224ce4b31131e61a01e426 P e4ed0f05ec9949a5a78562546d5566d2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 03:47:38.567687 16788 raft_consensus.cc:482] T 36771bd187224ce4b31131e61a01e426 P e4ed0f05ec9949a5a78562546d5566d2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 03:47:38.567798 16788 raft_consensus.cc:3037] T 36771bd187224ce4b31131e61a01e426 P e4ed0f05ec9949a5a78562546d5566d2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:47:38.570017 16788 raft_consensus.cc:504] T 36771bd187224ce4b31131e61a01e426 P e4ed0f05ec9949a5a78562546d5566d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5327dcf8cbc4adb804842b27d7ab913" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 41349 } } peers { permanent_uuid: "e4ed0f05ec9949a5a78562546d5566d2" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 41479 } } peers { permanent_uuid: "25395ab024974d1a88518d178c2e2ab2" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 40383 } }
I20260926 03:47:38.570719 16788 leader_election.cc:288] T 36771bd187224ce4b31131e61a01e426 P e4ed0f05ec9949a5a78562546d5566d2 [CANDIDATE]: Term 1 election: Requested vote from peers b5327dcf8cbc4adb804842b27d7ab913 (127.0.194.194:41349), 25395ab024974d1a88518d178c2e2ab2 (127.0.194.193:40383)
I20260926 03:47:38.571388 16449 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36771bd187224ce4b31131e61a01e426" candidate_uuid: "e4ed0f05ec9949a5a78562546d5566d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25395ab024974d1a88518d178c2e2ab2"
I20260926 03:47:38.571452 16583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36771bd187224ce4b31131e61a01e426" candidate_uuid: "e4ed0f05ec9949a5a78562546d5566d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5327dcf8cbc4adb804842b27d7ab913"
I20260926 03:47:38.571671 16449 raft_consensus.cc:3037] T 36771bd187224ce4b31131e61a01e426 P 25395ab024974d1a88518d178c2e2ab2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:47:38.571678 16583 raft_consensus.cc:3037] T 36771bd187224ce4b31131e61a01e426 P b5327dcf8cbc4adb804842b27d7ab913 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:47:38.574004 16449 raft_consensus.cc:2454] T 36771bd187224ce4b31131e61a01e426 P 25395ab024974d1a88518d178c2e2ab2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4ed0f05ec9949a5a78562546d5566d2 in term 1.
I20260926 03:47:38.574012 16583 raft_consensus.cc:2454] T 36771bd187224ce4b31131e61a01e426 P b5327dcf8cbc4adb804842b27d7ab913 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4ed0f05ec9949a5a78562546d5566d2 in term 1.
I20260926 03:47:38.574541 16650 leader_election.cc:302] T 36771bd187224ce4b31131e61a01e426 P e4ed0f05ec9949a5a78562546d5566d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 25395ab024974d1a88518d178c2e2ab2, e4ed0f05ec9949a5a78562546d5566d2; no voters:
I20260926 03:47:38.574898 16788 raft_consensus.cc:2781] T 36771bd187224ce4b31131e61a01e426 P e4ed0f05ec9949a5a78562546d5566d2 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:47:38.575379 16788 raft_consensus.cc:686] T 36771bd187224ce4b31131e61a01e426 P e4ed0f05ec9949a5a78562546d5566d2 [term 1 LEADER]: Becoming Leader. State: Replica: e4ed0f05ec9949a5a78562546d5566d2, State: Running, Role: LEADER
I20260926 03:47:38.575976 16788 consensus_queue.cc:237] T 36771bd187224ce4b31131e61a01e426 P e4ed0f05ec9949a5a78562546d5566d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5327dcf8cbc4adb804842b27d7ab913" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 41349 } } peers { permanent_uuid: "e4ed0f05ec9949a5a78562546d5566d2" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 41479 } } peers { permanent_uuid: "25395ab024974d1a88518d178c2e2ab2" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 40383 } }
I20260926 03:47:38.580766 16304 catalog_manager.cc:5495] T 36771bd187224ce4b31131e61a01e426 P e4ed0f05ec9949a5a78562546d5566d2 reported cstate change: term changed from 0 to 1, leader changed from <none> to e4ed0f05ec9949a5a78562546d5566d2 (127.0.194.195). New cstate: current_term: 1 leader_uuid: "e4ed0f05ec9949a5a78562546d5566d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5327dcf8cbc4adb804842b27d7ab913" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 41349 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4ed0f05ec9949a5a78562546d5566d2" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 41479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "25395ab024974d1a88518d178c2e2ab2" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 40383 } health_report { overall_health: UNKNOWN } } }
I20260926 03:47:38.645143 16779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:38.639237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43148 (local address 127.0.194.195:41479)
0926 03:47:38.639538 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0926 03:47:38.639547 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:47:38.639648 (+ 101us) server_negotiation.cc:407] Connection header received
0926 03:47:38.639922 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:47:38.639926 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:47:38.640015 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:47:38.640177 (+ 162us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:47:38.640592 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:47:38.641333 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:47:38.642802 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:47:38.643219 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:47:38.643351 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:47:38.643880 (+ 529us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:47:38.644530 (+ 650us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:47:38.644703 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0926 03:47:38.644772 (+ 69us) server_negotiation.cc:299] Negotiation successful
0926 03:47:38.644861 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170}
I20260926 03:47:38.653318 16583 raft_consensus.cc:1261] T 36771bd187224ce4b31131e61a01e426 P b5327dcf8cbc4adb804842b27d7ab913 [term 1 FOLLOWER]: Refusing update from remote peer e4ed0f05ec9949a5a78562546d5566d2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:47:38.653316 16449 raft_consensus.cc:1261] T 36771bd187224ce4b31131e61a01e426 P 25395ab024974d1a88518d178c2e2ab2 [term 1 FOLLOWER]: Refusing update from remote peer e4ed0f05ec9949a5a78562546d5566d2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:47:38.654409 16788 consensus_queue.cc:1035] T 36771bd187224ce4b31131e61a01e426 P e4ed0f05ec9949a5a78562546d5566d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5327dcf8cbc4adb804842b27d7ab913" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 41349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:47:38.654752 16790 consensus_queue.cc:1035] T 36771bd187224ce4b31131e61a01e426 P e4ed0f05ec9949a5a78562546d5566d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "25395ab024974d1a88518d178c2e2ab2" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 40383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:47:38.669301 16794 mvcc.cc:205] Tried to move back new op lower bound from 7333455702634311680 to 7333455702331211776. Current Snapshot: MvccSnapshot[applied={T|T < 7333455702634311680}]
I20260926 03:47:38.673986 16793 mvcc.cc:205] Tried to move back new op lower bound from 7333455702634311680 to 7333455702331211776. Current Snapshot: MvccSnapshot[applied={T|T < 7333455702634311680}]
I20260926 03:47:38.687740 16304 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52152:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:47:38.688071 16304 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52152:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:47:38.691766 16304 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 83b249b98dce4c2c86d27ac0720f4f75: Sending DeleteTablet for 3 replicas of tablet 36771bd187224ce4b31131e61a01e426
I20260926 03:47:38.693092 16696 tablet_service.cc:1514] Processing DeleteTablet for tablet 36771bd187224ce4b31131e61a01e426 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:47:38 UTC) from {username='kudu', principal='kudu/127.0.194.254@KRBTEST.COM'} at 127.0.0.1:43140
I20260926 03:47:38.693092 16429 tablet_service.cc:1514] Processing DeleteTablet for tablet 36771bd187224ce4b31131e61a01e426 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:47:38 UTC) from {username='kudu', principal='kudu/127.0.194.254@KRBTEST.COM'} at 127.0.0.1:39352
I20260926 03:47:38.693130 16563 tablet_service.cc:1514] Processing DeleteTablet for tablet 36771bd187224ce4b31131e61a01e426 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:47:38 UTC) from {username='kudu', principal='kudu/127.0.194.254@KRBTEST.COM'} at 127.0.0.1:48454
I20260926 03:47:38.693293 779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu pid 16366
I20260926 03:47:38.703398 16805 tablet_replica.cc:330] T 36771bd187224ce4b31131e61a01e426 P 25395ab024974d1a88518d178c2e2ab2: stopping tablet replica
I20260926 03:47:38.704048 16805 raft_consensus.cc:2229] T 36771bd187224ce4b31131e61a01e426 P 25395ab024974d1a88518d178c2e2ab2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:47:38.704775 16805 raft_consensus.cc:2258] T 36771bd187224ce4b31131e61a01e426 P 25395ab024974d1a88518d178c2e2ab2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:47:38.705639 16804 tablet_replica.cc:330] T 36771bd187224ce4b31131e61a01e426 P e4ed0f05ec9949a5a78562546d5566d2: stopping tablet replica
I20260926 03:47:38.706221 16804 raft_consensus.cc:2229] T 36771bd187224ce4b31131e61a01e426 P e4ed0f05ec9949a5a78562546d5566d2 [term 1 LEADER]: Raft consensus shutting down.
I20260926 03:47:38.707279 16804 raft_consensus.cc:2258] T 36771bd187224ce4b31131e61a01e426 P e4ed0f05ec9949a5a78562546d5566d2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:47:38.710021 16805 ts_tablet_manager.cc:1905] T 36771bd187224ce4b31131e61a01e426 P 25395ab024974d1a88518d178c2e2ab2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:47:38.711335 16806 tablet_replica.cc:330] T 36771bd187224ce4b31131e61a01e426 P b5327dcf8cbc4adb804842b27d7ab913: stopping tablet replica
I20260926 03:47:38.711722 16806 raft_consensus.cc:2229] T 36771bd187224ce4b31131e61a01e426 P b5327dcf8cbc4adb804842b27d7ab913 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:47:38.712527 16806 raft_consensus.cc:2258] T 36771bd187224ce4b31131e61a01e426 P b5327dcf8cbc4adb804842b27d7ab913 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:47:38.713485 16804 ts_tablet_manager.cc:1905] T 36771bd187224ce4b31131e61a01e426 P e4ed0f05ec9949a5a78562546d5566d2: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 26 03:47:38 dist-test-slave-n2th krb5kdc[16260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394455, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.194.193@KRBTEST.COM
I20260926 03:47:38.719035 16806 ts_tablet_manager.cc:1905] T 36771bd187224ce4b31131e61a01e426 P b5327dcf8cbc4adb804842b27d7ab913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:47:38.721861 16804 ts_tablet_manager.cc:1918] T 36771bd187224ce4b31131e61a01e426 P e4ed0f05ec9949a5a78562546d5566d2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:47:38.722144 16804 log.cc:1196] T 36771bd187224ce4b31131e61a01e426 P e4ed0f05ec9949a5a78562546d5566d2: Deleting WAL directory at /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/minicluster-data/ts-2/wal/wals/36771bd187224ce4b31131e61a01e426
I20260926 03:47:38.722860 16804 ts_tablet_manager.cc:1939] T 36771bd187224ce4b31131e61a01e426 P e4ed0f05ec9949a5a78562546d5566d2: Deleting consensus metadata
I20260926 03:47:38.724946 16291 catalog_manager.cc:4841] TS e4ed0f05ec9949a5a78562546d5566d2 (127.0.194.195:41479): tablet 36771bd187224ce4b31131e61a01e426 (table test-table [id=47c3a0bd5c1d4d63b606738006ef45ca]) successfully deleted
I20260926 03:47:38.726118 16777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:38.694780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39358 (local address 127.0.194.193:40383)
0926 03:47:38.695009 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0926 03:47:38.695018 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:47:38.702824 (+ 7806us) server_negotiation.cc:407] Connection header received
0926 03:47:38.704097 (+ 1273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:47:38.704104 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:47:38.704205 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:47:38.704291 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:47:38.707268 (+ 2977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:47:38.708460 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:47:38.710761 (+ 2301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:47:38.711409 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:47:38.711517 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:47:38.717868 (+ 6351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:47:38.717884 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:47:38.717905 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:47:38.717972 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:47:38.722133 (+ 4161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:47:38.723256 (+ 1123us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:47:38.723266 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:47:38.723270 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:47:38.723441 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:47:38.723923 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:47:38.723928 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:47:38.723931 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:47:38.724416 (+ 485us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:47:38.724712 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0926 03:47:38.725414 (+ 702us) server_negotiation.cc:299] Negotiation successful
0926 03:47:38.725665 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260926 03:47:38.726086 16806 ts_tablet_manager.cc:1918] T 36771bd187224ce4b31131e61a01e426 P b5327dcf8cbc4adb804842b27d7ab913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:47:38.726274 16806 log.cc:1196] T 36771bd187224ce4b31131e61a01e426 P b5327dcf8cbc4adb804842b27d7ab913: Deleting WAL directory at /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/minicluster-data/ts-1/wal/wals/36771bd187224ce4b31131e61a01e426
I20260926 03:47:38.727254 16489 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:47:38.727303 16806 ts_tablet_manager.cc:1939] T 36771bd187224ce4b31131e61a01e426 P b5327dcf8cbc4adb804842b27d7ab913: Deleting consensus metadata
I20260926 03:47:38.729475 16291 catalog_manager.cc:4841] TS b5327dcf8cbc4adb804842b27d7ab913 (127.0.194.194:41349): tablet 36771bd187224ce4b31131e61a01e426 (table test-table [id=47c3a0bd5c1d4d63b606738006ef45ca]) successfully deleted
I20260926 03:47:38.819958 16805 ts_tablet_manager.cc:1918] T 36771bd187224ce4b31131e61a01e426 P 25395ab024974d1a88518d178c2e2ab2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:47:38.820241 16805 log.cc:1196] T 36771bd187224ce4b31131e61a01e426 P 25395ab024974d1a88518d178c2e2ab2: Deleting WAL directory at /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/minicluster-data/ts-0/wal/wals/36771bd187224ce4b31131e61a01e426
I20260926 03:47:38.820729 779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu with pid 16366
W20260926 03:47:38.838352 16290 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.194.193:40383 (error 108)
I20260926 03:47:38.838653 779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu pid 16500
I20260926 03:47:38.840674 16810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:38.839826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.193:40383 (local address 127.0.0.1:39374)
0926 03:47:38.840314 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0926 03:47:38.840448 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.193:40383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":253,"threads_started":1}
W20260926 03:47:38.840960 16290 catalog_manager.cc:4584] TS 25395ab024974d1a88518d178c2e2ab2 (127.0.194.193:40383): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 36771bd187224ce4b31131e61a01e426: Network error: Client connection negotiation failed: client connection to 127.0.194.193:40383: connect: Connection refused (error 111)
Sep 26 03:47:38 dist-test-slave-n2th krb5kdc[16260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394455, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.194.194@KRBTEST.COM
I20260926 03:47:38.855499 16775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:38.839524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48462 (local address 127.0.194.194:41349)
0926 03:47:38.839735 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0926 03:47:38.839741 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:47:38.839791 (+ 50us) server_negotiation.cc:407] Connection header received
0926 03:47:38.840025 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:47:38.840035 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:47:38.840136 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:47:38.840244 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:47:38.841885 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:47:38.842593 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:47:38.844087 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:47:38.844487 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:47:38.844627 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:47:38.849415 (+ 4788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:47:38.849432 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:47:38.849448 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:47:38.849486 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:47:38.852489 (+ 3003us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:47:38.853200 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:47:38.853215 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:47:38.853217 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:47:38.853347 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:47:38.853734 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:47:38.853739 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:47:38.853743 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:47:38.854144 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:47:38.854453 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0926 03:47:38.854923 (+ 470us) server_negotiation.cc:299] Negotiation successful
0926 03:47:38.855144 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 03:47:38.856216 16623 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:47:38.894974 16810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:38.894351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.193:40383 (local address 127.0.0.1:39388)
0926 03:47:38.894629 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0926 03:47:38.894786 (+ 157us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.193:40383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113}
I20260926 03:47:38.896255 16810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:38.895800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.193:40383 (local address 127.0.0.1:39392)
0926 03:47:38.895991 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0926 03:47:38.896080 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.193:40383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260926 03:47:38.947422 779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu with pid 16500
I20260926 03:47:38.957094 16810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:38.956491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.193:40383 (local address 127.0.0.1:39402)
0926 03:47:38.956763 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0926 03:47:38.956884 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.193:40383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260926 03:47:38.958599 16810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:38.958112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.193:40383 (local address 127.0.0.1:39406)
0926 03:47:38.958311 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0926 03:47:38.958416 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.193:40383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260926 03:47:38.968024 779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu pid 16633
Sep 26 03:47:38 dist-test-slave-n2th krb5kdc[16260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394455, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.194.195@KRBTEST.COM
I20260926 03:47:38.987581 16779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:38.968869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43158 (local address 127.0.194.195:41479)
0926 03:47:38.969052 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0926 03:47:38.969062 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 03:47:38.969082 (+ 20us) server_negotiation.cc:407] Connection header received
0926 03:47:38.969275 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:47:38.969281 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:47:38.969381 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:47:38.969507 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:47:38.970919 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:47:38.971807 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:47:38.973850 (+ 2043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:47:38.974305 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:47:38.974403 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:47:38.980910 (+ 6507us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:47:38.980922 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:47:38.980938 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:47:38.980981 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:47:38.984219 (+ 3238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:47:38.985179 (+ 960us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:47:38.985187 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:47:38.985189 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:47:38.985367 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:47:38.985814 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:47:38.985817 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:47:38.985819 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:47:38.986286 (+ 467us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:47:38.986547 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0926 03:47:38.986996 (+ 449us) server_negotiation.cc:299] Negotiation successful
0926 03:47:38.987218 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 03:47:38.988337 16756 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:47:39.039315 16810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:39.038850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.193:40383 (local address 127.0.0.1:39408)
0926 03:47:39.039075 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0926 03:47:39.039161 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.193:40383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260926 03:47:39.040390 16810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:39.039986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.193:40383 (local address 127.0.0.1:39424)
0926 03:47:39.040185 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0926 03:47:39.040252 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.193:40383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260926 03:47:39.110910 779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu with pid 16633
I20260926 03:47:39.130614 779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu pid 16273
I20260926 03:47:39.131862 16335 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:47:39.243474 16810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:47:39.242806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.193:40383 (local address 127.0.0.1:39434)
0926 03:47:39.243089 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0926 03:47:39.243229 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.193:40383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260926 03:47:39.243705 779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu with pid 16273
I20260926 03:47:59.266145 779 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
/tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.194.254:41965
--webserver_interface=127.0.194.254
--webserver_port=44757
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/krb5kdc/kudu_127.0.194.254.keytab
--principal=kudu/127.0.194.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.194.212:43861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.194.254:41965
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:47:59.613870 16814 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:47:59.614295 16814 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:47:59.614463 16814 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:47:59.622126 16814 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260926 03:47:59.622282 16814 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260926 03:47:59.622387 16814 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260926 03:47:59.622485 16814 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:47:59.622589 16814 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:47:59.622707 16814 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:47:59.622825 16814 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:47:59.622907 16814 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:47:59.634399 16814 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.194.212:43861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.194.254:41965
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/krb5kdc/kudu_127.0.194.254.keytab
--principal=kudu/127.0.194.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.194.254:41965
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.194.254
--webserver_port=44757
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:47:59.635812 16814 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:47:59.637228 16814 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:47:59.661424 16823 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:47:59.661374 16820 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:47:59.661365 16821 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:47:59.662187 16814 server_base.cc:1034] running on GCE node
I20260926 03:47:59.663275 16814 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:47:59.665017 16814 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:47:59.666307 16814 hybrid_clock.cc:648] HybridClock initialized: now 1790394479666236 us; error 50 us; skew 500 ppm
I20260926 03:47:59.666838 16814 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:47:59 dist-test-slave-n2th krb5kdc[16260](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394479, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:47:59.670648 16814 init.cc:377] Logged in from keytab as kudu/127.0.194.254@KRBTEST.COM (short username kudu)
I20260926 03:47:59.672435 16814 webserver.cc:460] Webserver started at http://127.0.194.254:44757/ using document root <none> and password file <none>
I20260926 03:47:59.673292 16814 fs_manager.cc:359] Metadata directory not provided
I20260926 03:47:59.673454 16814 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:47:59.680469 16814 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 03:47:59.683941 16830 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:47:59.685458 16814 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 03:47:59.685679 16814 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/minicluster-data/master-0/data,/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/minicluster-data/master-0/wal
uuid: "83b249b98dce4c2c86d27ac0720f4f75"
format_stamp: "Formatted at 2026-09-26 03:47:36 on dist-test-slave-n2th"
server_key: "3264ade5582dda9c739fb0e857be8854"
server_key_iv: "8092c7d8338af7094eb3e0d4524d12c5"
server_key_version: "encryptionkey@0"
I20260926 03:47:59.686919 16814 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:47:59.706987 16814 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:47:59.711736 16814 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:47:59.712109 16814 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:47:59.735270 16814 rpc_server.cc:307] RPC server started. Bound to: 127.0.194.254:41965
I20260926 03:47:59.735311 16881 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.194.254:41965 every 8 connection(s)
I20260926 03:47:59.736912 16814 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/minicluster-data/master-0/data/info.pb
I20260926 03:47:59.741552 779 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu as pid 16814
I20260926 03:47:59.742462 779 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
/tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.194.193:40383
--local_ip_for_outbound_sockets=127.0.194.193
--tserver_master_addrs=127.0.194.254:41965
--webserver_port=39049
--webserver_interface=127.0.194.193
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/krb5kdc/kudu_127.0.194.193.keytab
--principal=kudu/127.0.194.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.194.212:43861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 03:47:59.744530 16882 sys_catalog.cc:263] Verifying existing consensus state
I20260926 03:47:59.748162 16882 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83b249b98dce4c2c86d27ac0720f4f75: Bootstrap starting.
I20260926 03:47:59.778439 16882 log.cc:824] T 00000000000000000000000000000000 P 83b249b98dce4c2c86d27ac0720f4f75: Log is configured to *not* fsync() on all Append() calls
I20260926 03:47:59.795173 16882 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83b249b98dce4c2c86d27ac0720f4f75: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 03:47:59.795878 16882 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83b249b98dce4c2c86d27ac0720f4f75: Bootstrap complete.
I20260926 03:47:59.810268 16882 raft_consensus.cc:348] T 00000000000000000000000000000000 P 83b249b98dce4c2c86d27ac0720f4f75 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83b249b98dce4c2c86d27ac0720f4f75" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 41965 } }
I20260926 03:47:59.811579 16882 raft_consensus.cc:729] T 00000000000000000000000000000000 P 83b249b98dce4c2c86d27ac0720f4f75 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83b249b98dce4c2c86d27ac0720f4f75, State: Initialized, Role: FOLLOWER
I20260926 03:47:59.812484 16882 consensus_queue.cc:260] T 00000000000000000000000000000000 P 83b249b98dce4c2c86d27ac0720f4f75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83b249b98dce4c2c86d27ac0720f4f75" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 41965 } }
I20260926 03:47:59.812852 16882 raft_consensus.cc:388] T 00000000000000000000000000000000 P 83b249b98dce4c2c86d27ac0720f4f75 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:47:59.813019 16882 raft_consensus.cc:482] T 00000000000000000000000000000000 P 83b249b98dce4c2c86d27ac0720f4f75 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:47:59.813269 16882 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 83b249b98dce4c2c86d27ac0720f4f75 [term 1 FOLLOWER]: Advancing to term 2
I20260926 03:47:59.818306 16882 raft_consensus.cc:504] T 00000000000000000000000000000000 P 83b249b98dce4c2c86d27ac0720f4f75 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83b249b98dce4c2c86d27ac0720f4f75" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 41965 } }
I20260926 03:47:59.819083 16882 leader_election.cc:302] T 00000000000000000000000000000000 P 83b249b98dce4c2c86d27ac0720f4f75 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 83b249b98dce4c2c86d27ac0720f4f75; no voters:
I20260926 03:47:59.819810 16882 leader_election.cc:288] T 00000000000000000000000000000000 P 83b249b98dce4c2c86d27ac0720f4f75 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 03:47:59.819969 16886 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 83b249b98dce4c2c86d27ac0720f4f75 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 03:47:59.821130 16886 raft_consensus.cc:686] T 00000000000000000000000000000000 P 83b249b98dce4c2c86d27ac0720f4f75 [term 2 LEADER]: Becoming Leader. State: Replica: 83b249b98dce4c2c86d27ac0720f4f75, State: Running, Role: LEADER
I20260926 03:47:59.821795 16886 consensus_queue.cc:237] T 00000000000000000000000000000000 P 83b249b98dce4c2c86d27ac0720f4f75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83b249b98dce4c2c86d27ac0720f4f75" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 41965 } }
I20260926 03:47:59.822275 16882 sys_catalog.cc:564] T 00000000000000000000000000000000 P 83b249b98dce4c2c86d27ac0720f4f75 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:47:59.824538 16887 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83b249b98dce4c2c86d27ac0720f4f75 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "83b249b98dce4c2c86d27ac0720f4f75" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83b249b98dce4c2c86d27ac0720f4f75" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 41965 } } }
I20260926 03:47:59.825002 16887 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83b249b98dce4c2c86d27ac0720f4f75 [sys.catalog]: This master's current role is: LEADER
I20260926 03:47:59.828845 16888 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83b249b98dce4c2c86d27ac0720f4f75 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 83b249b98dce4c2c86d27ac0720f4f75. Latest consensus state: current_term: 2 leader_uuid: "83b249b98dce4c2c86d27ac0720f4f75" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83b249b98dce4c2c86d27ac0720f4f75" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 41965 } } }
I20260926 03:47:59.829239 16888 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83b249b98dce4c2c86d27ac0720f4f75 [sys.catalog]: This master's current role is: LEADER
I20260926 03:47:59.831110 16892 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:47:59.841544 16892 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:47:59.843611 16892 catalog_manager.cc:1259] Loaded cluster ID: 8069980440c24e87975da30d412dedd2
I20260926 03:47:59.843783 16892 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:47:59.849388 16892 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:47:59.851976 16892 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 83b249b98dce4c2c86d27ac0720f4f75: Loaded TSK: 0
I20260926 03:47:59.885309 16892 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 83b249b98dce4c2c86d27ac0720f4f75: Generated new TSK 1
I20260926 03:47:59.885722 16892 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 03:48:00.155233 16884 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:48:00.155529 16884 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:48:00.155689 16884 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:48:00.163311 16884 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:48:00.163431 16884 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:48:00.163604 16884 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.194.193
I20260926 03:48:00.174011 16884 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.194.212:43861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/krb5kdc/kudu_127.0.194.193.keytab
--principal=kudu/127.0.194.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.194.193:40383
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.194.193
--webserver_port=39049
--tserver_master_addrs=127.0.194.254:41965
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.194.193
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:48:00.175495 16884 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:48:00.176942 16884 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:48:00.192592 16910 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:48:00.192900 16912 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:48:00.192605 16909 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:48:00.194368 16884 server_base.cc:1034] running on GCE node
I20260926 03:48:00.195173 16884 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:48:00.196316 16884 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:48:00.197573 16884 hybrid_clock.cc:648] HybridClock initialized: now 1790394480197505 us; error 64 us; skew 500 ppm
I20260926 03:48:00.197932 16884 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:48:00 dist-test-slave-n2th krb5kdc[16260](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394480, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:48:00.201313 16884 init.cc:377] Logged in from keytab as kudu/127.0.194.193@KRBTEST.COM (short username kudu)
I20260926 03:48:00.203095 16884 webserver.cc:460] Webserver started at http://127.0.194.193:39049/ using document root <none> and password file <none>
I20260926 03:48:00.203867 16884 fs_manager.cc:359] Metadata directory not provided
I20260926 03:48:00.204025 16884 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:48:00.209978 16884 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.006s
I20260926 03:48:00.213289 16919 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:48:00.214886 16884 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 03:48:00.215080 16884 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/minicluster-data/ts-0/wal
uuid: "25395ab024974d1a88518d178c2e2ab2"
format_stamp: "Formatted at 2026-09-26 03:47:37 on dist-test-slave-n2th"
server_key: "512e244a8f11792c6844c3e2977049c0"
server_key_iv: "80df830967f0818d4ef48c077dd0b700"
server_key_version: "encryptionkey@0"
I20260926 03:48:00.216197 16884 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:48:00.249048 16884 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:48:00.253533 16884 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:48:00.253836 16884 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:48:00.254954 16884 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:48:00.256836 16926 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
W20260926 03:48:00.263056 16926 ts_tablet_manager.cc:1852] T 36771bd187224ce4b31131e61a01e426 P 25395ab024974d1a88518d178c2e2ab2: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260926 03:48:00.264089 16926 ts_tablet_manager.cc:1905] T 36771bd187224ce4b31131e61a01e426 P 25395ab024974d1a88518d178c2e2ab2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:48:00.271760 16926 ts_tablet_manager.cc:1918] T 36771bd187224ce4b31131e61a01e426 P 25395ab024974d1a88518d178c2e2ab2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:48:00.272037 16926 ts_tablet_manager.cc:1939] T 36771bd187224ce4b31131e61a01e426 P 25395ab024974d1a88518d178c2e2ab2: Deleting consensus metadata
I20260926 03:48:00.272475 16884 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260926 03:48:00.272625 16884 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.016s user 0.001s sys 0.000s
I20260926 03:48:00.272773 16884 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:48:00.272862 16884 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:48:00.307866 16884 rpc_server.cc:307] RPC server started. Bound to: 127.0.194.193:40383
I20260926 03:48:00.307922 17032 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.194.193:40383 every 8 connection(s)
I20260926 03:48:00.309363 16884 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/minicluster-data/ts-0/data/info.pb
I20260926 03:48:00.313593 779 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu as pid 16884
I20260926 03:48:00.314327 779 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
/tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.194.194:41349
--local_ip_for_outbound_sockets=127.0.194.194
--tserver_master_addrs=127.0.194.254:41965
--webserver_port=37899
--webserver_interface=127.0.194.194
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/krb5kdc/kudu_127.0.194.194.keytab
--principal=kudu/127.0.194.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.194.212:43861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 03:48:00 dist-test-slave-n2th krb5kdc[16260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394480, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.193@KRBTEST.COM for kudu/127.0.194.254@KRBTEST.COM
I20260926 03:48:00.344372 17036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:00.312848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:41965 (local address 127.0.194.193:50877)
0926 03:48:00.313794 (+ 946us) negotiation.cc:107] Waiting for socket to connect
0926 03:48:00.313850 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0926 03:48:00.315839 (+ 1989us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:48:00.323180 (+ 7341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:48:00.323190 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:48:00.324017 (+ 827us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:48:00.324918 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:48:00.324966 (+ 48us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:48:00.326911 (+ 1945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:48:00.326920 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:48:00.329344 (+ 2424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:48:00.329367 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:48:00.330614 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:48:00.330638 (+ 24us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:48:00.331033 (+ 395us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:48:00.331915 (+ 882us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:48:00.331957 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:48:00.335265 (+ 3308us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:48:00.339527 (+ 4262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:48:00.339538 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:48:00.339556 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:48:00.340052 (+ 496us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:48:00.340572 (+ 520us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:48:00.340578 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:48:00.340581 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:48:00.340781 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:48:00.341613 (+ 832us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:48:00.341621 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:48:00.342018 (+ 397us) client_negotiation.cc:769] Sending connection context
0926 03:48:00.342307 (+ 289us) client_negotiation.cc:240] Negotiation successful
0926 03:48:00.342950 (+ 643us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":638,"thread_start_us":401,"threads_started":1}
I20260926 03:48:00.344360 17035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:00.312478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.193:50877 (local address 127.0.194.254:41965)
0926 03:48:00.313068 (+ 590us) server_negotiation.cc:206] Beginning negotiation
0926 03:48:00.313085 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 03:48:00.314027 (+ 942us) server_negotiation.cc:407] Connection header received
0926 03:48:00.316515 (+ 2488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:48:00.316536 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:48:00.317068 (+ 532us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:48:00.322619 (+ 5551us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:48:00.325255 (+ 2636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:48:00.326655 (+ 1400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:48:00.329588 (+ 2933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:48:00.330316 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:48:00.330467 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:48:00.335769 (+ 5302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:48:00.335784 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:48:00.335810 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:48:00.335903 (+ 93us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:48:00.339330 (+ 3427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:48:00.340234 (+ 904us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:48:00.340242 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:48:00.340249 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:48:00.340432 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:48:00.340920 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:48:00.340924 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:48:00.340927 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:48:00.341452 (+ 525us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:48:00.341660 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0926 03:48:00.342550 (+ 890us) server_negotiation.cc:299] Negotiation successful
0926 03:48:00.342999 (+ 449us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":235,"threads_started":1}
I20260926 03:48:00.348042 17033 heartbeater.cc:347] Connected to a master server at 127.0.194.254:41965
I20260926 03:48:00.348381 17033 heartbeater.cc:464] Registering TS with master...
I20260926 03:48:00.349210 17033 heartbeater.cc:511] Master 127.0.194.254:41965 requested a full tablet report, sending...
I20260926 03:48:00.351769 16847 ts_manager.cc:194] Registered new tserver with Master: 25395ab024974d1a88518d178c2e2ab2 (127.0.194.193:40383)
I20260926 03:48:00.353768 16847 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.194.193@KRBTEST.COM'} at 127.0.194.193:50877
W20260926 03:48:00.667228 17037 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:48:00.667531 17037 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:48:00.667629 17037 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:48:00.675081 17037 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:48:00.675209 17037 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:48:00.675330 17037 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.194.194
I20260926 03:48:00.685593 17037 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.194.212:43861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/krb5kdc/kudu_127.0.194.194.keytab
--principal=kudu/127.0.194.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.194.194:41349
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.194.194
--webserver_port=37899
--tserver_master_addrs=127.0.194.254:41965
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.194.194
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:48:00.686951 17037 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:48:00.688241 17037 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:48:00.701736 17042 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:48:00.701718 17043 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:48:00.701803 17045 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:48:00.702638 17037 server_base.cc:1034] running on GCE node
I20260926 03:48:00.703424 17037 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:48:00.704443 17037 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:48:00.705687 17037 hybrid_clock.cc:648] HybridClock initialized: now 1790394480705613 us; error 67 us; skew 500 ppm
I20260926 03:48:00.706044 17037 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:48:00 dist-test-slave-n2th krb5kdc[16260](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394480, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:48:00.709539 17037 init.cc:377] Logged in from keytab as kudu/127.0.194.194@KRBTEST.COM (short username kudu)
I20260926 03:48:00.711143 17037 webserver.cc:460] Webserver started at http://127.0.194.194:37899/ using document root <none> and password file <none>
I20260926 03:48:00.711881 17037 fs_manager.cc:359] Metadata directory not provided
I20260926 03:48:00.712042 17037 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:48:00.717914 17037 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.003s
I20260926 03:48:00.721239 17052 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:48:00.722982 17037 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 03:48:00.723181 17037 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/minicluster-data/ts-1/wal
uuid: "b5327dcf8cbc4adb804842b27d7ab913"
format_stamp: "Formatted at 2026-09-26 03:47:37 on dist-test-slave-n2th"
server_key: "6001f474c3d8de06bca8de24bdb8b31d"
server_key_iv: "308b140ceb06480085467a7e42793253"
server_key_version: "encryptionkey@0"
I20260926 03:48:00.724282 17037 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:48:00.745507 17037 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:48:00.751057 17037 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:48:00.751365 17037 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:48:00.752655 17037 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:48:00.754477 17037 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:48:00.754590 17037 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:48:00.754686 17037 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:48:00.754779 17037 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:48:00.789614 17037 rpc_server.cc:307] RPC server started. Bound to: 127.0.194.194:41349
I20260926 03:48:00.789758 17164 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.194.194:41349 every 8 connection(s)
I20260926 03:48:00.791370 17037 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/minicluster-data/ts-1/data/info.pb
I20260926 03:48:00.794528 779 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu as pid 17037
I20260926 03:48:00.795390 779 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
/tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.194.195:41479
--local_ip_for_outbound_sockets=127.0.194.195
--tserver_master_addrs=127.0.194.254:41965
--webserver_port=35835
--webserver_interface=127.0.194.195
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/krb5kdc/kudu_127.0.194.195.keytab
--principal=kudu/127.0.194.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.194.212:43861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 03:48:00 dist-test-slave-n2th krb5kdc[16260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394480, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.194@KRBTEST.COM for kudu/127.0.194.254@KRBTEST.COM
I20260926 03:48:00.816587 17035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:00.794311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.194:60019 (local address 127.0.194.254:41965)
0926 03:48:00.794529 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0926 03:48:00.794536 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:48:00.795797 (+ 1261us) server_negotiation.cc:407] Connection header received
0926 03:48:00.798835 (+ 3038us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:48:00.798841 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:48:00.798929 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:48:00.799032 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:48:00.801270 (+ 2238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:48:00.801984 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:48:00.804407 (+ 2423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:48:00.805173 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:48:00.805331 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:48:00.809275 (+ 3944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:48:00.809285 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:48:00.809291 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:48:00.809363 (+ 72us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:48:00.813050 (+ 3687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:48:00.813877 (+ 827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:48:00.813885 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:48:00.813892 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:48:00.814047 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:48:00.814475 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:48:00.814483 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:48:00.814489 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:48:00.814977 (+ 488us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:48:00.815129 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0926 03:48:00.815856 (+ 727us) server_negotiation.cc:299] Negotiation successful
0926 03:48:00.816144 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260926 03:48:00.817528 17167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:00.794684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:41965 (local address 127.0.194.194:60019)
0926 03:48:00.795603 (+ 919us) negotiation.cc:107] Waiting for socket to connect
0926 03:48:00.795659 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0926 03:48:00.798459 (+ 2800us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:48:00.799337 (+ 878us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:48:00.799352 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:48:00.800183 (+ 831us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:48:00.801053 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:48:00.801082 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:48:00.802170 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:48:00.802187 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:48:00.804232 (+ 2045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:48:00.804245 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:48:00.805403 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:48:00.805408 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:48:00.805649 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:48:00.806399 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:48:00.806423 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:48:00.808992 (+ 2569us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:48:00.813228 (+ 4236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:48:00.813236 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:48:00.813252 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:48:00.813716 (+ 464us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:48:00.814158 (+ 442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:48:00.814161 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:48:00.814164 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:48:00.814350 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:48:00.815111 (+ 761us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:48:00.815118 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:48:00.815470 (+ 352us) client_negotiation.cc:769] Sending connection context
0926 03:48:00.815704 (+ 234us) client_negotiation.cc:240] Negotiation successful
0926 03:48:00.816142 (+ 438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":589,"thread_start_us":304,"threads_started":1}
I20260926 03:48:00.819464 17165 heartbeater.cc:347] Connected to a master server at 127.0.194.254:41965
I20260926 03:48:00.819927 17165 heartbeater.cc:464] Registering TS with master...
I20260926 03:48:00.820811 17165 heartbeater.cc:511] Master 127.0.194.254:41965 requested a full tablet report, sending...
I20260926 03:48:00.822665 16847 ts_manager.cc:194] Registered new tserver with Master: b5327dcf8cbc4adb804842b27d7ab913 (127.0.194.194:41349)
I20260926 03:48:00.824075 16847 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.194.194@KRBTEST.COM'} at 127.0.194.194:60019
W20260926 03:48:01.139812 17168 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:48:01.140107 17168 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:48:01.140208 17168 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:48:01.147505 17168 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:48:01.147639 17168 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:48:01.147758 17168 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.194.195
I20260926 03:48:01.157671 17168 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.194.212:43861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/krb5kdc/kudu_127.0.194.195.keytab
--principal=kudu/127.0.194.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.194.195:41479
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.194.195
--webserver_port=35835
--tserver_master_addrs=127.0.194.254:41965
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.194.195
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:48:01.158955 17168 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:48:01.160241 17168 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:48:01.173547 17173 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:48:01.173465 17176 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:48:01.173521 17174 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:48:01.174189 17168 server_base.cc:1034] running on GCE node
I20260926 03:48:01.174927 17168 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:48:01.175851 17168 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:48:01.177065 17168 hybrid_clock.cc:648] HybridClock initialized: now 1790394481177020 us; error 31 us; skew 500 ppm
I20260926 03:48:01.177419 17168 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:48:01 dist-test-slave-n2th krb5kdc[16260](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394481, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:48:01.180708 17168 init.cc:377] Logged in from keytab as kudu/127.0.194.195@KRBTEST.COM (short username kudu)
I20260926 03:48:01.182178 17168 webserver.cc:460] Webserver started at http://127.0.194.195:35835/ using document root <none> and password file <none>
I20260926 03:48:01.182969 17168 fs_manager.cc:359] Metadata directory not provided
I20260926 03:48:01.183130 17168 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:48:01.188828 17168 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.004s
I20260926 03:48:01.191864 17183 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:48:01.193403 17168 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 03:48:01.193548 17168 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/minicluster-data/ts-2/wal
uuid: "e4ed0f05ec9949a5a78562546d5566d2"
format_stamp: "Formatted at 2026-09-26 03:47:38 on dist-test-slave-n2th"
server_key: "2e953a21f430b59eee8dcee318951152"
server_key_iv: "5324ade121b9e7de9d7c29bf0d51e50e"
server_key_version: "encryptionkey@0"
I20260926 03:48:01.194617 17168 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:48:01.222616 17168 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:48:01.227634 17168 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:48:01.227959 17168 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:48:01.229115 17168 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:48:01.230557 17168 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:48:01.230674 17168 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:48:01.230806 17168 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:48:01.230904 17168 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:48:01.265683 17168 rpc_server.cc:307] RPC server started. Bound to: 127.0.194.195:41479
I20260926 03:48:01.265786 17295 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.194.195:41479 every 8 connection(s)
I20260926 03:48:01.267313 17168 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/minicluster-data/ts-2/data/info.pb
I20260926 03:48:01.275628 779 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu as pid 17168
Sep 26 03:48:01 dist-test-slave-n2th krb5kdc[16260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394481, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.195@KRBTEST.COM for kudu/127.0.194.254@KRBTEST.COM
I20260926 03:48:01.288309 17035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:01.269938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.195:54601 (local address 127.0.194.254:41965)
0926 03:48:01.270160 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0926 03:48:01.270166 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:48:01.271394 (+ 1228us) server_negotiation.cc:407] Connection header received
0926 03:48:01.273193 (+ 1799us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:48:01.273197 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:48:01.273268 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:48:01.273361 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:48:01.274941 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:48:01.275675 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:48:01.277807 (+ 2132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:48:01.278303 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:48:01.278449 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:48:01.281997 (+ 3548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:48:01.282002 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:48:01.282005 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:48:01.282038 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:48:01.284766 (+ 2728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:48:01.285586 (+ 820us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:48:01.285590 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:48:01.285592 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:48:01.285757 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:48:01.286383 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:48:01.286387 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:48:01.286389 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:48:01.286697 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:48:01.286907 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0926 03:48:01.287821 (+ 914us) server_negotiation.cc:299] Negotiation successful
0926 03:48:01.288033 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260926 03:48:01.289644 17298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:01.270484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:41965 (local address 127.0.194.195:54601)
0926 03:48:01.271224 (+ 740us) negotiation.cc:107] Waiting for socket to connect
0926 03:48:01.271268 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 03:48:01.272930 (+ 1662us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:48:01.273574 (+ 644us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:48:01.273582 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:48:01.274134 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:48:01.274704 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:48:01.274721 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:48:01.275861 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:48:01.275866 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:48:01.277651 (+ 1785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:48:01.277665 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:48:01.278562 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:48:01.278569 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:48:01.278871 (+ 302us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:48:01.279569 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:48:01.279592 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:48:01.281762 (+ 2170us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:48:01.284941 (+ 3179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:48:01.284952 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:48:01.284971 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:48:01.285444 (+ 473us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:48:01.285908 (+ 464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:48:01.285916 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:48:01.285920 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:48:01.286248 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:48:01.286875 (+ 627us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:48:01.286881 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:48:01.287394 (+ 513us) client_negotiation.cc:769] Sending connection context
0926 03:48:01.287730 (+ 336us) client_negotiation.cc:240] Negotiation successful
0926 03:48:01.288358 (+ 628us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":327,"threads_started":1}
I20260926 03:48:01.291343 17299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:01.276586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49528 (local address 127.0.194.254:41965)
0926 03:48:01.277120 (+ 534us) server_negotiation.cc:206] Beginning negotiation
0926 03:48:01.277127 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:48:01.277457 (+ 330us) server_negotiation.cc:407] Connection header received
0926 03:48:01.277663 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:48:01.277668 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:48:01.277729 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:48:01.277849 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:48:01.279461 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:48:01.280197 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:48:01.281640 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:48:01.282109 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:48:01.282255 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:48:01.285210 (+ 2955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:48:01.285215 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:48:01.285218 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:48:01.285248 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:48:01.288524 (+ 3276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:48:01.289229 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:48:01.289234 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:48:01.289238 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:48:01.289378 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:48:01.289747 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:48:01.289752 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:48:01.289754 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:48:01.290091 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:48:01.290347 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0926 03:48:01.290751 (+ 404us) server_negotiation.cc:299] Negotiation successful
0926 03:48:01.290982 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":406,"thread_start_us":276,"threads_started":1}
I20260926 03:48:01.291416 17296 heartbeater.cc:347] Connected to a master server at 127.0.194.254:41965
I20260926 03:48:01.291831 17296 heartbeater.cc:464] Registering TS with master...
I20260926 03:48:01.292742 17296 heartbeater.cc:511] Master 127.0.194.254:41965 requested a full tablet report, sending...
I20260926 03:48:01.294269 16847 ts_manager.cc:194] Registered new tserver with Master: e4ed0f05ec9949a5a78562546d5566d2 (127.0.194.195:41479)
I20260926 03:48:01.295171 16847 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.194.195@KRBTEST.COM'} at 127.0.194.195:54601
I20260926 03:48:01.306512 779 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 03:48:01.313654 17299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:01.307663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49534 (local address 127.0.194.254:41965)
0926 03:48:01.307850 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0926 03:48:01.307860 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 03:48:01.308266 (+ 406us) server_negotiation.cc:407] Connection header received
0926 03:48:01.308548 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:48:01.308552 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:48:01.308696 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:48:01.308858 (+ 162us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:48:01.309300 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:48:01.310012 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:48:01.311643 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:48:01.312104 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:48:01.312243 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:48:01.312764 (+ 521us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:48:01.312962 (+ 198us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0926 03:48:01.313360 (+ 398us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:49534: token expired
Metrics: {"server-negotiator.queue_time_us":56}
W20260926 03:48:01.313758 17299 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:49534: token expired
W20260926 03:48:01.314281 16766 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.194.254:41965)
I20260926 03:48:01.334645 17299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:01.314846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49550 (local address 127.0.194.254:41965)
0926 03:48:01.314991 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0926 03:48:01.314994 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 03:48:01.315009 (+ 15us) server_negotiation.cc:407] Connection header received
0926 03:48:01.315075 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:48:01.315078 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:48:01.315131 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:48:01.315207 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:48:01.316593 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:48:01.317370 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:48:01.318988 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:48:01.319416 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:48:01.319512 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:48:01.328544 (+ 9032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:48:01.328551 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:48:01.328555 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:48:01.328601 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:48:01.331954 (+ 3353us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:48:01.332852 (+ 898us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:48:01.332857 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:48:01.332859 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:48:01.333013 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:48:01.333382 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:48:01.333386 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:48:01.333388 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:48:01.333725 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:48:01.333928 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0926 03:48:01.334209 (+ 281us) server_negotiation.cc:299] Negotiation successful
0926 03:48:01.334392 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 03:48:01.338212 16847 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49550:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 03:48:01.346201 16847 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 03:48:01.357411 17033 heartbeater.cc:503] Master 127.0.194.254:41965 was elected leader, sending a full tablet report...
I20260926 03:48:01.374115 17307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:01.359099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59050 (local address 127.0.194.194:41349)
0926 03:48:01.359846 (+ 747us) server_negotiation.cc:206] Beginning negotiation
0926 03:48:01.359853 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:48:01.360708 (+ 855us) server_negotiation.cc:407] Connection header received
0926 03:48:01.361198 (+ 490us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:48:01.361210 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:48:01.361556 (+ 346us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:48:01.361755 (+ 199us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:48:01.363058 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:48:01.365353 (+ 2295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:48:01.370457 (+ 5104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:48:01.372801 (+ 2344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:48:01.372997 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:48:01.373272 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0926 03:48:01.373606 (+ 334us) server_negotiation.cc:299] Negotiation successful
0926 03:48:01.373809 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":587,"thread_start_us":481,"threads_started":1}
I20260926 03:48:01.375016 17306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:01.358797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.194:41349 (local address 127.0.0.1:59050)
0926 03:48:01.360515 (+ 1718us) negotiation.cc:107] Waiting for socket to connect
0926 03:48:01.360568 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 03:48:01.360911 (+ 343us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:48:01.362171 (+ 1260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:48:01.362182 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:48:01.362224 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:48:01.362818 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:48:01.362835 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:48:01.365606 (+ 2771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:48:01.365615 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:48:01.370187 (+ 4572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:48:01.370217 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:48:01.373077 (+ 2860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:48:01.373086 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:48:01.373306 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:48:01.373376 (+ 70us) client_negotiation.cc:769] Sending connection context
0926 03:48:01.374323 (+ 947us) client_negotiation.cc:240] Negotiation successful
0926 03:48:01.374541 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1434,"thread_start_us":1260,"threads_started":1}
I20260926 03:48:01.379505 17311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:01.361846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.195:41479 (local address 127.0.0.1:43446)
0926 03:48:01.371123 (+ 9277us) negotiation.cc:107] Waiting for socket to connect
0926 03:48:01.371140 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 03:48:01.371249 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:48:01.372566 (+ 1317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:48:01.372573 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:48:01.372599 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:48:01.372941 (+ 342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:48:01.372956 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:48:01.374888 (+ 1932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:48:01.374897 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:48:01.376854 (+ 1957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:48:01.376872 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:48:01.378495 (+ 1623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:48:01.378501 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:48:01.378788 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:48:01.378830 (+ 42us) client_negotiation.cc:769] Sending connection context
0926 03:48:01.378941 (+ 111us) client_negotiation.cc:240] Negotiation successful
0926 03:48:01.379096 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9163,"mutex_wait_us":5,"thread_start_us":9037,"threads_started":1}
I20260926 03:48:01.379503 17310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:01.361445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43446 (local address 127.0.194.195:41479)
0926 03:48:01.371851 (+ 10406us) server_negotiation.cc:206] Beginning negotiation
0926 03:48:01.371860 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:48:01.371912 (+ 52us) server_negotiation.cc:407] Connection header received
0926 03:48:01.372012 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:48:01.372019 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:48:01.372308 (+ 289us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:48:01.372514 (+ 206us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:48:01.373190 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:48:01.374762 (+ 1572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:48:01.377044 (+ 2282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:48:01.378123 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:48:01.378418 (+ 295us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:48:01.378679 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0926 03:48:01.378988 (+ 309us) server_negotiation.cc:299] Negotiation successful
0926 03:48:01.379154 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10218,"thread_start_us":10439,"threads_started":1}
I20260926 03:48:01.379411 17100 tablet_service.cc:1467] Processing CreateTablet for tablet 78d97cb2f75840b596a4a0166c3b67a0 (DEFAULT_TABLE table=test-table [id=9710b8ca0196401d905c11d5453732d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:48:01.380983 17100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78d97cb2f75840b596a4a0166c3b67a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:48:01.382396 17308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:01.359556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.193:40383 (local address 127.0.0.1:60046)
0926 03:48:01.370215 (+ 10659us) negotiation.cc:107] Waiting for socket to connect
0926 03:48:01.370241 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0926 03:48:01.370395 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:48:01.374355 (+ 3960us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:48:01.374362 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:48:01.374394 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:48:01.374802 (+ 408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:48:01.374814 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:48:01.377183 (+ 2369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:48:01.377188 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:48:01.379180 (+ 1992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:48:01.379203 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:48:01.381556 (+ 2353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:48:01.381563 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:48:01.381756 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:48:01.381796 (+ 40us) client_negotiation.cc:769] Sending connection context
0926 03:48:01.381891 (+ 95us) client_negotiation.cc:240] Negotiation successful
0926 03:48:01.382022 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10462,"mutex_wait_us":29,"thread_start_us":10926,"threads_started":1}
I20260926 03:48:01.382476 17309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:01.360207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60046 (local address 127.0.194.193:40383)
0926 03:48:01.373823 (+ 13616us) server_negotiation.cc:206] Beginning negotiation
0926 03:48:01.373829 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:48:01.373854 (+ 25us) server_negotiation.cc:407] Connection header received
0926 03:48:01.373929 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:48:01.373934 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:48:01.374112 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:48:01.374986 (+ 874us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:48:01.375225 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:48:01.377044 (+ 1819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:48:01.379394 (+ 2350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:48:01.381327 (+ 1933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:48:01.381472 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:48:01.381683 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0926 03:48:01.381978 (+ 295us) server_negotiation.cc:299] Negotiation successful
0926 03:48:01.382142 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13412,"thread_start_us":13281,"threads_started":1}
I20260926 03:48:01.383677 17231 tablet_service.cc:1467] Processing CreateTablet for tablet 78d97cb2f75840b596a4a0166c3b67a0 (DEFAULT_TABLE table=test-table [id=9710b8ca0196401d905c11d5453732d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:48:01.384960 17231 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78d97cb2f75840b596a4a0166c3b67a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:48:01.384759 16968 tablet_service.cc:1467] Processing CreateTablet for tablet 78d97cb2f75840b596a4a0166c3b67a0 (DEFAULT_TABLE table=test-table [id=9710b8ca0196401d905c11d5453732d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:48:01.386152 16968 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78d97cb2f75840b596a4a0166c3b67a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:48:01.396910 17312 tablet_bootstrap.cc:492] T 78d97cb2f75840b596a4a0166c3b67a0 P 25395ab024974d1a88518d178c2e2ab2: Bootstrap starting.
I20260926 03:48:01.399721 17314 tablet_bootstrap.cc:492] T 78d97cb2f75840b596a4a0166c3b67a0 P e4ed0f05ec9949a5a78562546d5566d2: Bootstrap starting.
I20260926 03:48:01.401181 17312 tablet_bootstrap.cc:654] T 78d97cb2f75840b596a4a0166c3b67a0 P 25395ab024974d1a88518d178c2e2ab2: Neither blocks nor log segments found. Creating new log.
I20260926 03:48:01.402500 17312 log.cc:824] T 78d97cb2f75840b596a4a0166c3b67a0 P 25395ab024974d1a88518d178c2e2ab2: Log is configured to *not* fsync() on all Append() calls
I20260926 03:48:01.403367 17313 tablet_bootstrap.cc:492] T 78d97cb2f75840b596a4a0166c3b67a0 P b5327dcf8cbc4adb804842b27d7ab913: Bootstrap starting.
I20260926 03:48:01.404264 17314 tablet_bootstrap.cc:654] T 78d97cb2f75840b596a4a0166c3b67a0 P e4ed0f05ec9949a5a78562546d5566d2: Neither blocks nor log segments found. Creating new log.
I20260926 03:48:01.405678 17314 log.cc:824] T 78d97cb2f75840b596a4a0166c3b67a0 P e4ed0f05ec9949a5a78562546d5566d2: Log is configured to *not* fsync() on all Append() calls
I20260926 03:48:01.407364 17313 tablet_bootstrap.cc:654] T 78d97cb2f75840b596a4a0166c3b67a0 P b5327dcf8cbc4adb804842b27d7ab913: Neither blocks nor log segments found. Creating new log.
I20260926 03:48:01.408710 17313 log.cc:824] T 78d97cb2f75840b596a4a0166c3b67a0 P b5327dcf8cbc4adb804842b27d7ab913: Log is configured to *not* fsync() on all Append() calls
I20260926 03:48:01.409185 17314 tablet_bootstrap.cc:492] T 78d97cb2f75840b596a4a0166c3b67a0 P e4ed0f05ec9949a5a78562546d5566d2: No bootstrap required, opened a new log
I20260926 03:48:01.409519 17312 tablet_bootstrap.cc:492] T 78d97cb2f75840b596a4a0166c3b67a0 P 25395ab024974d1a88518d178c2e2ab2: No bootstrap required, opened a new log
I20260926 03:48:01.409509 17314 ts_tablet_manager.cc:1397] T 78d97cb2f75840b596a4a0166c3b67a0 P e4ed0f05ec9949a5a78562546d5566d2: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20260926 03:48:01.409864 17312 ts_tablet_manager.cc:1397] T 78d97cb2f75840b596a4a0166c3b67a0 P 25395ab024974d1a88518d178c2e2ab2: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.005s
I20260926 03:48:01.412617 17313 tablet_bootstrap.cc:492] T 78d97cb2f75840b596a4a0166c3b67a0 P b5327dcf8cbc4adb804842b27d7ab913: No bootstrap required, opened a new log
I20260926 03:48:01.412915 17313 ts_tablet_manager.cc:1397] T 78d97cb2f75840b596a4a0166c3b67a0 P b5327dcf8cbc4adb804842b27d7ab913: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.003s
I20260926 03:48:01.417945 17314 raft_consensus.cc:348] T 78d97cb2f75840b596a4a0166c3b67a0 P e4ed0f05ec9949a5a78562546d5566d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5327dcf8cbc4adb804842b27d7ab913" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 41349 } } peers { permanent_uuid: "e4ed0f05ec9949a5a78562546d5566d2" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 41479 } } peers { permanent_uuid: "25395ab024974d1a88518d178c2e2ab2" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 40383 } }
I20260926 03:48:01.418385 17314 raft_consensus.cc:374] T 78d97cb2f75840b596a4a0166c3b67a0 P e4ed0f05ec9949a5a78562546d5566d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:48:01.418488 17314 raft_consensus.cc:729] T 78d97cb2f75840b596a4a0166c3b67a0 P e4ed0f05ec9949a5a78562546d5566d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4ed0f05ec9949a5a78562546d5566d2, State: Initialized, Role: FOLLOWER
I20260926 03:48:01.419239 17314 consensus_queue.cc:260] T 78d97cb2f75840b596a4a0166c3b67a0 P e4ed0f05ec9949a5a78562546d5566d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5327dcf8cbc4adb804842b27d7ab913" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 41349 } } peers { permanent_uuid: "e4ed0f05ec9949a5a78562546d5566d2" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 41479 } } peers { permanent_uuid: "25395ab024974d1a88518d178c2e2ab2" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 40383 } }
I20260926 03:48:01.421527 17296 heartbeater.cc:503] Master 127.0.194.254:41965 was elected leader, sending a full tablet report...
I20260926 03:48:01.422252 17314 ts_tablet_manager.cc:1428] T 78d97cb2f75840b596a4a0166c3b67a0 P e4ed0f05ec9949a5a78562546d5566d2: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260926 03:48:01.421028 17312 raft_consensus.cc:348] T 78d97cb2f75840b596a4a0166c3b67a0 P 25395ab024974d1a88518d178c2e2ab2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5327dcf8cbc4adb804842b27d7ab913" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 41349 } } peers { permanent_uuid: "e4ed0f05ec9949a5a78562546d5566d2" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 41479 } } peers { permanent_uuid: "25395ab024974d1a88518d178c2e2ab2" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 40383 } }
I20260926 03:48:01.423100 17312 raft_consensus.cc:374] T 78d97cb2f75840b596a4a0166c3b67a0 P 25395ab024974d1a88518d178c2e2ab2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:48:01.423219 17312 raft_consensus.cc:729] T 78d97cb2f75840b596a4a0166c3b67a0 P 25395ab024974d1a88518d178c2e2ab2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25395ab024974d1a88518d178c2e2ab2, State: Initialized, Role: FOLLOWER
I20260926 03:48:01.423975 17312 consensus_queue.cc:260] T 78d97cb2f75840b596a4a0166c3b67a0 P 25395ab024974d1a88518d178c2e2ab2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5327dcf8cbc4adb804842b27d7ab913" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 41349 } } peers { permanent_uuid: "e4ed0f05ec9949a5a78562546d5566d2" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 41479 } } peers { permanent_uuid: "25395ab024974d1a88518d178c2e2ab2" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 40383 } }
I20260926 03:48:01.423897 17313 raft_consensus.cc:348] T 78d97cb2f75840b596a4a0166c3b67a0 P b5327dcf8cbc4adb804842b27d7ab913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5327dcf8cbc4adb804842b27d7ab913" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 41349 } } peers { permanent_uuid: "e4ed0f05ec9949a5a78562546d5566d2" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 41479 } } peers { permanent_uuid: "25395ab024974d1a88518d178c2e2ab2" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 40383 } }
I20260926 03:48:01.424352 17313 raft_consensus.cc:374] T 78d97cb2f75840b596a4a0166c3b67a0 P b5327dcf8cbc4adb804842b27d7ab913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:48:01.424458 17313 raft_consensus.cc:729] T 78d97cb2f75840b596a4a0166c3b67a0 P b5327dcf8cbc4adb804842b27d7ab913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5327dcf8cbc4adb804842b27d7ab913, State: Initialized, Role: FOLLOWER
I20260926 03:48:01.425845 17312 ts_tablet_manager.cc:1428] T 78d97cb2f75840b596a4a0166c3b67a0 P 25395ab024974d1a88518d178c2e2ab2: Time spent starting tablet: real 0.016s user 0.012s sys 0.003s
I20260926 03:48:01.426533 17313 consensus_queue.cc:260] T 78d97cb2f75840b596a4a0166c3b67a0 P b5327dcf8cbc4adb804842b27d7ab913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5327dcf8cbc4adb804842b27d7ab913" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 41349 } } peers { permanent_uuid: "e4ed0f05ec9949a5a78562546d5566d2" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 41479 } } peers { permanent_uuid: "25395ab024974d1a88518d178c2e2ab2" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 40383 } }
I20260926 03:48:01.427845 17165 heartbeater.cc:503] Master 127.0.194.254:41965 was elected leader, sending a full tablet report...
I20260926 03:48:01.428653 17313 ts_tablet_manager.cc:1428] T 78d97cb2f75840b596a4a0166c3b67a0 P b5327dcf8cbc4adb804842b27d7ab913: Time spent starting tablet: real 0.016s user 0.015s sys 0.000s
I20260926 03:48:01.482834 17320 raft_consensus.cc:482] T 78d97cb2f75840b596a4a0166c3b67a0 P b5327dcf8cbc4adb804842b27d7ab913 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 03:48:01.483175 17320 raft_consensus.cc:504] T 78d97cb2f75840b596a4a0166c3b67a0 P b5327dcf8cbc4adb804842b27d7ab913 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5327dcf8cbc4adb804842b27d7ab913" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 41349 } } peers { permanent_uuid: "e4ed0f05ec9949a5a78562546d5566d2" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 41479 } } peers { permanent_uuid: "25395ab024974d1a88518d178c2e2ab2" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 40383 } }
I20260926 03:48:01.484933 17320 leader_election.cc:288] T 78d97cb2f75840b596a4a0166c3b67a0 P b5327dcf8cbc4adb804842b27d7ab913 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e4ed0f05ec9949a5a78562546d5566d2 (127.0.194.195:41479), 25395ab024974d1a88518d178c2e2ab2 (127.0.194.193:40383)
I20260926 03:48:01.494364 17310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:01.485403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.194:34563 (local address 127.0.194.195:41479)
0926 03:48:01.485611 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0926 03:48:01.485617 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:48:01.486372 (+ 755us) server_negotiation.cc:407] Connection header received
0926 03:48:01.486565 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:48:01.486571 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:48:01.486668 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:48:01.486812 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:48:01.487682 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:48:01.488757 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:48:01.491765 (+ 3008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:48:01.493090 (+ 1325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:48:01.493259 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:48:01.493339 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 03:48:01.493954 (+ 615us) server_negotiation.cc:299] Negotiation successful
0926 03:48:01.494091 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 03:48:01.494356 17309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:01.485503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.194:47531 (local address 127.0.194.193:40383)
0926 03:48:01.485757 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0926 03:48:01.485765 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:48:01.486372 (+ 607us) server_negotiation.cc:407] Connection header received
0926 03:48:01.486565 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:48:01.486571 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:48:01.486668 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:48:01.486801 (+ 133us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:48:01.487620 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:48:01.488757 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:48:01.491789 (+ 3032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:48:01.493088 (+ 1299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:48:01.493259 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:48:01.493339 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 03:48:01.493954 (+ 615us) server_negotiation.cc:299] Negotiation successful
0926 03:48:01.494091 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 03:48:01.494611 17321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:01.485341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.195:41479 (local address 127.0.194.194:34563)
0926 03:48:01.486251 (+ 910us) negotiation.cc:107] Waiting for socket to connect
0926 03:48:01.486268 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 03:48:01.486426 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:48:01.486819 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:48:01.486824 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:48:01.486865 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:48:01.487516 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:48:01.487529 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:48:01.488926 (+ 1397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:48:01.488932 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:48:01.491572 (+ 2640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:48:01.491589 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:48:01.493384 (+ 1795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:48:01.493391 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:48:01.493683 (+ 292us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:48:01.493795 (+ 112us) client_negotiation.cc:769] Sending connection context
0926 03:48:01.493892 (+ 97us) client_negotiation.cc:240] Negotiation successful
0926 03:48:01.494088 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":790,"mutex_wait_us":319,"spinlock_wait_cycles":55808,"thread_start_us":430,"threads_started":1}
I20260926 03:48:01.495105 17322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:01.485225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.193:40383 (local address 127.0.194.194:47531)
0926 03:48:01.486191 (+ 966us) negotiation.cc:107] Waiting for socket to connect
0926 03:48:01.486219 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0926 03:48:01.486393 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:48:01.486819 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:48:01.486824 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:48:01.486855 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:48:01.487410 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:48:01.487426 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:48:01.488927 (+ 1501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:48:01.488932 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:48:01.491573 (+ 2641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:48:01.491589 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:48:01.493384 (+ 1795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:48:01.493391 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:48:01.493685 (+ 294us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:48:01.493791 (+ 106us) client_negotiation.cc:769] Sending connection context
0926 03:48:01.493892 (+ 101us) client_negotiation.cc:240] Negotiation successful
0926 03:48:01.494088 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":811,"mutex_wait_us":280,"thread_start_us":695,"threads_started":1}
I20260926 03:48:01.495781 17251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78d97cb2f75840b596a4a0166c3b67a0" candidate_uuid: "b5327dcf8cbc4adb804842b27d7ab913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4ed0f05ec9949a5a78562546d5566d2" is_pre_election: true
I20260926 03:48:01.496156 16988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78d97cb2f75840b596a4a0166c3b67a0" candidate_uuid: "b5327dcf8cbc4adb804842b27d7ab913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25395ab024974d1a88518d178c2e2ab2" is_pre_election: true
I20260926 03:48:01.496275 17251 raft_consensus.cc:2454] T 78d97cb2f75840b596a4a0166c3b67a0 P e4ed0f05ec9949a5a78562546d5566d2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b5327dcf8cbc4adb804842b27d7ab913 in term 0.
I20260926 03:48:01.496665 16988 raft_consensus.cc:2454] T 78d97cb2f75840b596a4a0166c3b67a0 P 25395ab024974d1a88518d178c2e2ab2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b5327dcf8cbc4adb804842b27d7ab913 in term 0.
I20260926 03:48:01.497046 17055 leader_election.cc:302] T 78d97cb2f75840b596a4a0166c3b67a0 P b5327dcf8cbc4adb804842b27d7ab913 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b5327dcf8cbc4adb804842b27d7ab913, e4ed0f05ec9949a5a78562546d5566d2; no voters:
I20260926 03:48:01.497489 17320 raft_consensus.cc:2781] T 78d97cb2f75840b596a4a0166c3b67a0 P b5327dcf8cbc4adb804842b27d7ab913 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 03:48:01.497635 17320 raft_consensus.cc:482] T 78d97cb2f75840b596a4a0166c3b67a0 P b5327dcf8cbc4adb804842b27d7ab913 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 03:48:01.497735 17320 raft_consensus.cc:3037] T 78d97cb2f75840b596a4a0166c3b67a0 P b5327dcf8cbc4adb804842b27d7ab913 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:48:01.500208 17320 raft_consensus.cc:504] T 78d97cb2f75840b596a4a0166c3b67a0 P b5327dcf8cbc4adb804842b27d7ab913 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5327dcf8cbc4adb804842b27d7ab913" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 41349 } } peers { permanent_uuid: "e4ed0f05ec9949a5a78562546d5566d2" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 41479 } } peers { permanent_uuid: "25395ab024974d1a88518d178c2e2ab2" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 40383 } }
I20260926 03:48:01.500828 17320 leader_election.cc:288] T 78d97cb2f75840b596a4a0166c3b67a0 P b5327dcf8cbc4adb804842b27d7ab913 [CANDIDATE]: Term 1 election: Requested vote from peers e4ed0f05ec9949a5a78562546d5566d2 (127.0.194.195:41479), 25395ab024974d1a88518d178c2e2ab2 (127.0.194.193:40383)
I20260926 03:48:01.501386 16988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78d97cb2f75840b596a4a0166c3b67a0" candidate_uuid: "b5327dcf8cbc4adb804842b27d7ab913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25395ab024974d1a88518d178c2e2ab2"
I20260926 03:48:01.501386 17251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78d97cb2f75840b596a4a0166c3b67a0" candidate_uuid: "b5327dcf8cbc4adb804842b27d7ab913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4ed0f05ec9949a5a78562546d5566d2"
I20260926 03:48:01.501605 17251 raft_consensus.cc:3037] T 78d97cb2f75840b596a4a0166c3b67a0 P e4ed0f05ec9949a5a78562546d5566d2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:48:01.501605 16988 raft_consensus.cc:3037] T 78d97cb2f75840b596a4a0166c3b67a0 P 25395ab024974d1a88518d178c2e2ab2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:48:01.504945 16988 raft_consensus.cc:2454] T 78d97cb2f75840b596a4a0166c3b67a0 P 25395ab024974d1a88518d178c2e2ab2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b5327dcf8cbc4adb804842b27d7ab913 in term 1.
I20260926 03:48:01.504945 17251 raft_consensus.cc:2454] T 78d97cb2f75840b596a4a0166c3b67a0 P e4ed0f05ec9949a5a78562546d5566d2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b5327dcf8cbc4adb804842b27d7ab913 in term 1.
I20260926 03:48:01.505412 17055 leader_election.cc:302] T 78d97cb2f75840b596a4a0166c3b67a0 P b5327dcf8cbc4adb804842b27d7ab913 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 25395ab024974d1a88518d178c2e2ab2, b5327dcf8cbc4adb804842b27d7ab913, e4ed0f05ec9949a5a78562546d5566d2; no voters:
I20260926 03:48:01.505776 17320 raft_consensus.cc:2781] T 78d97cb2f75840b596a4a0166c3b67a0 P b5327dcf8cbc4adb804842b27d7ab913 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:48:01.506227 17320 raft_consensus.cc:686] T 78d97cb2f75840b596a4a0166c3b67a0 P b5327dcf8cbc4adb804842b27d7ab913 [term 1 LEADER]: Becoming Leader. State: Replica: b5327dcf8cbc4adb804842b27d7ab913, State: Running, Role: LEADER
I20260926 03:48:01.506721 17320 consensus_queue.cc:237] T 78d97cb2f75840b596a4a0166c3b67a0 P b5327dcf8cbc4adb804842b27d7ab913 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5327dcf8cbc4adb804842b27d7ab913" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 41349 } } peers { permanent_uuid: "e4ed0f05ec9949a5a78562546d5566d2" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 41479 } } peers { permanent_uuid: "25395ab024974d1a88518d178c2e2ab2" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 40383 } }
I20260926 03:48:01.511620 16847 catalog_manager.cc:5495] T 78d97cb2f75840b596a4a0166c3b67a0 P b5327dcf8cbc4adb804842b27d7ab913 reported cstate change: term changed from 0 to 1, leader changed from <none> to b5327dcf8cbc4adb804842b27d7ab913 (127.0.194.194). New cstate: current_term: 1 leader_uuid: "b5327dcf8cbc4adb804842b27d7ab913" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5327dcf8cbc4adb804842b27d7ab913" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 41349 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4ed0f05ec9949a5a78562546d5566d2" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 41479 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "25395ab024974d1a88518d178c2e2ab2" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 40383 } health_report { overall_health: UNKNOWN } } }
W20260926 03:48:01.518975 17297 tablet.cc:2369] T 78d97cb2f75840b596a4a0166c3b67a0 P e4ed0f05ec9949a5a78562546d5566d2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 03:48:01.543537 17166 tablet.cc:2369] T 78d97cb2f75840b596a4a0166c3b67a0 P b5327dcf8cbc4adb804842b27d7ab913: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 03:48:01.548615 17307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:01.542687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59054 (local address 127.0.194.194:41349)
0926 03:48:01.542898 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0926 03:48:01.542907 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:48:01.542952 (+ 45us) server_negotiation.cc:407] Connection header received
0926 03:48:01.543218 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:48:01.543244 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:48:01.543375 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:48:01.543538 (+ 163us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:48:01.543967 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:48:01.544868 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:48:01.546280 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:48:01.546677 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:48:01.546848 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:48:01.547367 (+ 519us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:48:01.547925 (+ 558us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:48:01.548042 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0926 03:48:01.548153 (+ 111us) server_negotiation.cc:299] Negotiation successful
0926 03:48:01.548390 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 03:48:01.557466 16988 raft_consensus.cc:1261] T 78d97cb2f75840b596a4a0166c3b67a0 P 25395ab024974d1a88518d178c2e2ab2 [term 1 FOLLOWER]: Refusing update from remote peer b5327dcf8cbc4adb804842b27d7ab913: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:48:01.557472 17251 raft_consensus.cc:1261] T 78d97cb2f75840b596a4a0166c3b67a0 P e4ed0f05ec9949a5a78562546d5566d2 [term 1 FOLLOWER]: Refusing update from remote peer b5327dcf8cbc4adb804842b27d7ab913: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:48:01.558583 17323 consensus_queue.cc:1035] T 78d97cb2f75840b596a4a0166c3b67a0 P b5327dcf8cbc4adb804842b27d7ab913 [LEADER]: Connected to new peer: Peer: permanent_uuid: "25395ab024974d1a88518d178c2e2ab2" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 40383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:48:01.559011 17320 consensus_queue.cc:1035] T 78d97cb2f75840b596a4a0166c3b67a0 P b5327dcf8cbc4adb804842b27d7ab913 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4ed0f05ec9949a5a78562546d5566d2" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 41479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 03:48:01.561921 17034 tablet.cc:2369] T 78d97cb2f75840b596a4a0166c3b67a0 P 25395ab024974d1a88518d178c2e2ab2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 03:48:01.572397 17326 mvcc.cc:205] Tried to move back new op lower bound from 7333455796447649792 to 7333455796255547392. Current Snapshot: MvccSnapshot[applied={T|T < 7333455796447649792}]
I20260926 03:48:01.592749 16847 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49550:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:48:01.593053 16847 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49550:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:48:01.597009 16847 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 83b249b98dce4c2c86d27ac0720f4f75: Sending DeleteTablet for 3 replicas of tablet 78d97cb2f75840b596a4a0166c3b67a0
I20260926 03:48:01.598297 17100 tablet_service.cc:1514] Processing DeleteTablet for tablet 78d97cb2f75840b596a4a0166c3b67a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:48:01 UTC) from {username='kudu', principal='kudu/127.0.194.254@KRBTEST.COM'} at 127.0.0.1:59050
I20260926 03:48:01.598382 17231 tablet_service.cc:1514] Processing DeleteTablet for tablet 78d97cb2f75840b596a4a0166c3b67a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:48:01 UTC) from {username='kudu', principal='kudu/127.0.194.254@KRBTEST.COM'} at 127.0.0.1:43446
I20260926 03:48:01.598596 16968 tablet_service.cc:1514] Processing DeleteTablet for tablet 78d97cb2f75840b596a4a0166c3b67a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:48:01 UTC) from {username='kudu', principal='kudu/127.0.194.254@KRBTEST.COM'} at 127.0.0.1:60046
I20260926 03:48:01.602931 779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu pid 16884
I20260926 03:48:01.612308 17338 tablet_replica.cc:330] T 78d97cb2f75840b596a4a0166c3b67a0 P 25395ab024974d1a88518d178c2e2ab2: stopping tablet replica
I20260926 03:48:01.612926 17338 raft_consensus.cc:2229] T 78d97cb2f75840b596a4a0166c3b67a0 P 25395ab024974d1a88518d178c2e2ab2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:48:01.613574 17338 raft_consensus.cc:2258] T 78d97cb2f75840b596a4a0166c3b67a0 P 25395ab024974d1a88518d178c2e2ab2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:48:01.616576 17338 ts_tablet_manager.cc:1905] T 78d97cb2f75840b596a4a0166c3b67a0 P 25395ab024974d1a88518d178c2e2ab2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:48:01.624250 17309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:01.604336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60056 (local address 127.0.194.193:40383)
0926 03:48:01.604575 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0926 03:48:01.604584 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:48:01.604610 (+ 26us) server_negotiation.cc:407] Connection header received
0926 03:48:01.604688 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:48:01.604696 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:48:01.604810 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:48:01.604924 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:48:01.606808 (+ 1884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:48:01.608483 (+ 1675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:48:01.610465 (+ 1982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:48:01.611210 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:48:01.611347 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:48:01.615225 (+ 3878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:48:01.615243 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:48:01.615264 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:48:01.615318 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:48:01.619571 (+ 4253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:48:01.620688 (+ 1117us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:48:01.620698 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:48:01.620701 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:48:01.620968 (+ 267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:48:01.621458 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:48:01.621463 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:48:01.621468 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:48:01.621939 (+ 471us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:48:01.622306 (+ 367us) server_negotiation.cc:1035] Waiting for connection context
0926 03:48:01.623519 (+ 1213us) server_negotiation.cc:299] Negotiation successful
0926 03:48:01.623799 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 03:48:01.625262 17028 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:48:01.628402 17337 tablet_replica.cc:330] T 78d97cb2f75840b596a4a0166c3b67a0 P e4ed0f05ec9949a5a78562546d5566d2: stopping tablet replica
I20260926 03:48:01.629603 17337 raft_consensus.cc:2229] T 78d97cb2f75840b596a4a0166c3b67a0 P e4ed0f05ec9949a5a78562546d5566d2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:48:01.630179 17337 raft_consensus.cc:2258] T 78d97cb2f75840b596a4a0166c3b67a0 P e4ed0f05ec9949a5a78562546d5566d2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:48:01.631251 17336 tablet_replica.cc:330] T 78d97cb2f75840b596a4a0166c3b67a0 P b5327dcf8cbc4adb804842b27d7ab913: stopping tablet replica
I20260926 03:48:01.632112 17336 raft_consensus.cc:2229] T 78d97cb2f75840b596a4a0166c3b67a0 P b5327dcf8cbc4adb804842b27d7ab913 [term 1 LEADER]: Raft consensus shutting down.
I20260926 03:48:01.632933 17336 raft_consensus.cc:2258] T 78d97cb2f75840b596a4a0166c3b67a0 P b5327dcf8cbc4adb804842b27d7ab913 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:48:01.637471 17337 ts_tablet_manager.cc:1905] T 78d97cb2f75840b596a4a0166c3b67a0 P e4ed0f05ec9949a5a78562546d5566d2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:48:01.638792 17336 ts_tablet_manager.cc:1905] T 78d97cb2f75840b596a4a0166c3b67a0 P b5327dcf8cbc4adb804842b27d7ab913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:48:01.644659 17337 ts_tablet_manager.cc:1918] T 78d97cb2f75840b596a4a0166c3b67a0 P e4ed0f05ec9949a5a78562546d5566d2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:48:01.644907 17337 log.cc:1196] T 78d97cb2f75840b596a4a0166c3b67a0 P e4ed0f05ec9949a5a78562546d5566d2: Deleting WAL directory at /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/minicluster-data/ts-2/wal/wals/78d97cb2f75840b596a4a0166c3b67a0
I20260926 03:48:01.645525 17337 ts_tablet_manager.cc:1939] T 78d97cb2f75840b596a4a0166c3b67a0 P e4ed0f05ec9949a5a78562546d5566d2: Deleting consensus metadata
I20260926 03:48:01.646432 17336 ts_tablet_manager.cc:1918] T 78d97cb2f75840b596a4a0166c3b67a0 P b5327dcf8cbc4adb804842b27d7ab913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:48:01.646662 17336 log.cc:1196] T 78d97cb2f75840b596a4a0166c3b67a0 P b5327dcf8cbc4adb804842b27d7ab913: Deleting WAL directory at /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/minicluster-data/ts-1/wal/wals/78d97cb2f75840b596a4a0166c3b67a0
I20260926 03:48:01.647336 17336 ts_tablet_manager.cc:1939] T 78d97cb2f75840b596a4a0166c3b67a0 P b5327dcf8cbc4adb804842b27d7ab913: Deleting consensus metadata
I20260926 03:48:01.647912 16833 catalog_manager.cc:4841] TS e4ed0f05ec9949a5a78562546d5566d2 (127.0.194.195:41479): tablet 78d97cb2f75840b596a4a0166c3b67a0 (table test-table [id=9710b8ca0196401d905c11d5453732d9]) successfully deleted
I20260926 03:48:01.649055 16833 catalog_manager.cc:4841] TS b5327dcf8cbc4adb804842b27d7ab913 (127.0.194.194:41349): tablet 78d97cb2f75840b596a4a0166c3b67a0 (table test-table [id=9710b8ca0196401d905c11d5453732d9]) successfully deleted
I20260926 03:48:01.739688 17338 ts_tablet_manager.cc:1918] T 78d97cb2f75840b596a4a0166c3b67a0 P 25395ab024974d1a88518d178c2e2ab2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:48:01.739950 17338 log.cc:1196] T 78d97cb2f75840b596a4a0166c3b67a0 P 25395ab024974d1a88518d178c2e2ab2: Deleting WAL directory at /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/minicluster-data/ts-0/wal/wals/78d97cb2f75840b596a4a0166c3b67a0
I20260926 03:48:01.740355 779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu with pid 16884
W20260926 03:48:01.764518 16832 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.194.193:40383 (error 108)
I20260926 03:48:01.765931 779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu pid 17037
I20260926 03:48:01.766546 17308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:01.765981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.193:40383 (local address 127.0.0.1:60066)
0926 03:48:01.766254 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0926 03:48:01.766379 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.193:40383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140}
W20260926 03:48:01.767014 16832 catalog_manager.cc:4584] TS 25395ab024974d1a88518d178c2e2ab2 (127.0.194.193:40383): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 78d97cb2f75840b596a4a0166c3b67a0: Network error: Client connection negotiation failed: client connection to 127.0.194.193:40383: connect: Connection refused (error 111)
I20260926 03:48:01.787287 17307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:01.766950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59064 (local address 127.0.194.194:41349)
0926 03:48:01.767137 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0926 03:48:01.767143 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:48:01.767192 (+ 49us) server_negotiation.cc:407] Connection header received
0926 03:48:01.767260 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:48:01.767265 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:48:01.767365 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:48:01.767451 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:48:01.769167 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:48:01.770077 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:48:01.772173 (+ 2096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:48:01.772646 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:48:01.772758 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:48:01.781041 (+ 8283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:48:01.781058 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:48:01.781074 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:48:01.781118 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:48:01.784285 (+ 3167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:48:01.785180 (+ 895us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:48:01.785188 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:48:01.785190 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:48:01.785333 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:48:01.785733 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:48:01.785736 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:48:01.785738 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:48:01.786092 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:48:01.786345 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0926 03:48:01.786674 (+ 329us) server_negotiation.cc:299] Negotiation successful
0926 03:48:01.786908 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 03:48:01.788038 17160 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:48:01.820767 17308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:01.820305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.193:40383 (local address 127.0.0.1:60082)
0926 03:48:01.820515 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0926 03:48:01.820625 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.193:40383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260926 03:48:01.821754 17308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:01.821458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.193:40383 (local address 127.0.0.1:60090)
0926 03:48:01.821578 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0926 03:48:01.821657 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.193:40383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260926 03:48:01.882308 17308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:01.881839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.193:40383 (local address 127.0.0.1:60106)
0926 03:48:01.882056 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0926 03:48:01.882155 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.193:40383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260926 03:48:01.883479 17308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:01.883118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.193:40383 (local address 127.0.0.1:60114)
0926 03:48:01.883282 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0926 03:48:01.883354 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.193:40383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260926 03:48:01.901944 779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu with pid 17037
I20260926 03:48:01.920627 779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu pid 17168
I20260926 03:48:01.933987 17310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:01.921373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43460 (local address 127.0.194.195:41479)
0926 03:48:01.921540 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0926 03:48:01.921544 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:48:01.921562 (+ 18us) server_negotiation.cc:407] Connection header received
0926 03:48:01.921729 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:48:01.921733 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:48:01.921800 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:48:01.921899 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:48:01.923313 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:48:01.924040 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:48:01.925303 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:48:01.925680 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:48:01.925770 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:48:01.928496 (+ 2726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:48:01.928510 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:48:01.928527 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:48:01.928570 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:48:01.931453 (+ 2883us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:48:01.932209 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:48:01.932216 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:48:01.932218 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:48:01.932372 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:48:01.932731 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:48:01.932734 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:48:01.932736 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:48:01.933071 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:48:01.933274 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0926 03:48:01.933568 (+ 294us) server_negotiation.cc:299] Negotiation successful
0926 03:48:01.933728 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 03:48:01.934638 17291 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:48:01.964210 17308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:01.963631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.193:40383 (local address 127.0.0.1:60124)
0926 03:48:01.963879 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0926 03:48:01.963996 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.193:40383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260926 03:48:01.965420 17308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:01.964939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.193:40383 (local address 127.0.0.1:60130)
0926 03:48:01.965178 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0926 03:48:01.965269 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.193:40383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129}
I20260926 03:48:02.023948 779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu with pid 17168
I20260926 03:48:02.040592 779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu pid 16814
I20260926 03:48:02.041735 16877 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:48:02.158048 17308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:02.157252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.193:40383 (local address 127.0.0.1:60134)
0926 03:48:02.157437 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0926 03:48:02.157616 (+ 179us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.193:40383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260926 03:48:02.158185 779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu with pid 16814
I20260926 03:48:22.177781 779 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
/tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.194.254:41965
--webserver_interface=127.0.194.254
--webserver_port=44757
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/krb5kdc/kudu_127.0.194.254.keytab
--principal=kudu/127.0.194.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.194.212:43861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.194.254:41965
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:48:22.517917 17344 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:48:22.518282 17344 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:48:22.518445 17344 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:48:22.525744 17344 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260926 03:48:22.525909 17344 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260926 03:48:22.526007 17344 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260926 03:48:22.526105 17344 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:48:22.526218 17344 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:48:22.526321 17344 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:48:22.526413 17344 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:48:22.526496 17344 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:48:22.536520 17344 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.194.212:43861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.194.254:41965
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/krb5kdc/kudu_127.0.194.254.keytab
--principal=kudu/127.0.194.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.194.254:41965
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.194.254
--webserver_port=44757
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:48:22.537887 17344 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:48:22.539306 17344 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:48:22.551396 17349 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:48:22.551477 17352 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:48:22.551460 17350 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:48:22.552677 17344 server_base.cc:1034] running on GCE node
I20260926 03:48:22.553570 17344 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:48:22.555110 17344 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:48:22.556373 17344 hybrid_clock.cc:648] HybridClock initialized: now 1790394502556282 us; error 78 us; skew 500 ppm
I20260926 03:48:22.556751 17344 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:48:22 dist-test-slave-n2th krb5kdc[16260](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394502, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:48:22.566828 17344 init.cc:377] Logged in from keytab as kudu/127.0.194.254@KRBTEST.COM (short username kudu)
I20260926 03:48:22.568531 17344 webserver.cc:460] Webserver started at http://127.0.194.254:44757/ using document root <none> and password file <none>
I20260926 03:48:22.569344 17344 fs_manager.cc:359] Metadata directory not provided
I20260926 03:48:22.569490 17344 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:48:22.575526 17344 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.002s
I20260926 03:48:22.578593 17359 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:48:22.580114 17344 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 03:48:22.580317 17344 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/minicluster-data/master-0/data,/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/minicluster-data/master-0/wal
uuid: "83b249b98dce4c2c86d27ac0720f4f75"
format_stamp: "Formatted at 2026-09-26 03:47:36 on dist-test-slave-n2th"
server_key: "3264ade5582dda9c739fb0e857be8854"
server_key_iv: "8092c7d8338af7094eb3e0d4524d12c5"
server_key_version: "encryptionkey@0"
I20260926 03:48:22.581327 17344 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:48:22.603331 17344 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:48:22.607970 17344 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:48:22.608275 17344 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:48:22.627586 17344 rpc_server.cc:307] RPC server started. Bound to: 127.0.194.254:41965
I20260926 03:48:22.627621 17410 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.194.254:41965 every 8 connection(s)
I20260926 03:48:22.629200 17344 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/minicluster-data/master-0/data/info.pb
I20260926 03:48:22.632246 779 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu as pid 17344
I20260926 03:48:22.633082 779 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
/tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.194.193:40383
--local_ip_for_outbound_sockets=127.0.194.193
--tserver_master_addrs=127.0.194.254:41965
--webserver_port=39049
--webserver_interface=127.0.194.193
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/krb5kdc/kudu_127.0.194.193.keytab
--principal=kudu/127.0.194.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.194.212:43861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 03:48:22.636497 17411 sys_catalog.cc:263] Verifying existing consensus state
I20260926 03:48:22.639834 17411 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83b249b98dce4c2c86d27ac0720f4f75: Bootstrap starting.
I20260926 03:48:22.671273 17411 log.cc:824] T 00000000000000000000000000000000 P 83b249b98dce4c2c86d27ac0720f4f75: Log is configured to *not* fsync() on all Append() calls
I20260926 03:48:22.692619 17411 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83b249b98dce4c2c86d27ac0720f4f75: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 03:48:22.693398 17411 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83b249b98dce4c2c86d27ac0720f4f75: Bootstrap complete.
I20260926 03:48:22.708798 17411 raft_consensus.cc:348] T 00000000000000000000000000000000 P 83b249b98dce4c2c86d27ac0720f4f75 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83b249b98dce4c2c86d27ac0720f4f75" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 41965 } }
I20260926 03:48:22.710115 17411 raft_consensus.cc:729] T 00000000000000000000000000000000 P 83b249b98dce4c2c86d27ac0720f4f75 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83b249b98dce4c2c86d27ac0720f4f75, State: Initialized, Role: FOLLOWER
I20260926 03:48:22.710929 17411 consensus_queue.cc:260] T 00000000000000000000000000000000 P 83b249b98dce4c2c86d27ac0720f4f75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83b249b98dce4c2c86d27ac0720f4f75" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 41965 } }
I20260926 03:48:22.711207 17411 raft_consensus.cc:388] T 00000000000000000000000000000000 P 83b249b98dce4c2c86d27ac0720f4f75 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:48:22.711328 17411 raft_consensus.cc:482] T 00000000000000000000000000000000 P 83b249b98dce4c2c86d27ac0720f4f75 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:48:22.711509 17411 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 83b249b98dce4c2c86d27ac0720f4f75 [term 2 FOLLOWER]: Advancing to term 3
I20260926 03:48:22.717190 17411 raft_consensus.cc:504] T 00000000000000000000000000000000 P 83b249b98dce4c2c86d27ac0720f4f75 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83b249b98dce4c2c86d27ac0720f4f75" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 41965 } }
I20260926 03:48:22.717823 17411 leader_election.cc:302] T 00000000000000000000000000000000 P 83b249b98dce4c2c86d27ac0720f4f75 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 83b249b98dce4c2c86d27ac0720f4f75; no voters:
I20260926 03:48:22.718535 17411 leader_election.cc:288] T 00000000000000000000000000000000 P 83b249b98dce4c2c86d27ac0720f4f75 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260926 03:48:22.718693 17415 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 83b249b98dce4c2c86d27ac0720f4f75 [term 3 FOLLOWER]: Leader election won for term 3
I20260926 03:48:22.720714 17411 sys_catalog.cc:564] T 00000000000000000000000000000000 P 83b249b98dce4c2c86d27ac0720f4f75 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:48:22.725451 17415 raft_consensus.cc:686] T 00000000000000000000000000000000 P 83b249b98dce4c2c86d27ac0720f4f75 [term 3 LEADER]: Becoming Leader. State: Replica: 83b249b98dce4c2c86d27ac0720f4f75, State: Running, Role: LEADER
I20260926 03:48:22.726159 17415 consensus_queue.cc:237] T 00000000000000000000000000000000 P 83b249b98dce4c2c86d27ac0720f4f75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83b249b98dce4c2c86d27ac0720f4f75" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 41965 } }
I20260926 03:48:22.729071 17417 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83b249b98dce4c2c86d27ac0720f4f75 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 83b249b98dce4c2c86d27ac0720f4f75. Latest consensus state: current_term: 3 leader_uuid: "83b249b98dce4c2c86d27ac0720f4f75" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83b249b98dce4c2c86d27ac0720f4f75" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 41965 } } }
I20260926 03:48:22.729072 17416 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83b249b98dce4c2c86d27ac0720f4f75 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "83b249b98dce4c2c86d27ac0720f4f75" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83b249b98dce4c2c86d27ac0720f4f75" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 41965 } } }
I20260926 03:48:22.729523 17417 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83b249b98dce4c2c86d27ac0720f4f75 [sys.catalog]: This master's current role is: LEADER
I20260926 03:48:22.729609 17416 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83b249b98dce4c2c86d27ac0720f4f75 [sys.catalog]: This master's current role is: LEADER
I20260926 03:48:22.743887 17425 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:48:22.753607 17425 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:48:22.755394 17425 catalog_manager.cc:1259] Loaded cluster ID: 8069980440c24e87975da30d412dedd2
I20260926 03:48:22.755568 17425 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:48:22.759775 17425 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:48:22.761548 17425 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 83b249b98dce4c2c86d27ac0720f4f75: Loaded TSK: 0
I20260926 03:48:22.761787 17425 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 83b249b98dce4c2c86d27ac0720f4f75: Loaded TSK: 1
F20260926 03:48:22.763032 17425 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fa423e349ad google::LogMessage::Fail() at ??:0
@ 0x7fa423e3788a google::LogMessage::SendToLog() at ??:0
@ 0x7fa423e344de google::LogMessage::Flush() at ??:0
@ 0x7fa423e35a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fa427b987d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fa427b8773e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7fa4398631c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7fa439862783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7fa4398ee2a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7fa4398ee0c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7fa4399385e9 std::function<>::operator()() at ??:0
@ 0x7fa439866cdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7fa439864c36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fa4398ec610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7fa4398ec439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fa426a80345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fa426a8391c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7fa426a83799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fa426a5eeaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7fa429d386db start_thread at ??:0
@ 0x7fa422cce61f clone at ??:0
W20260926 03:48:23.084548 17413 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:48:23.084849 17413 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:48:23.084949 17413 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:48:23.092561 17413 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:48:23.092689 17413 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:48:23.092806 17413 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.194.193
I20260926 03:48:23.103778 17413 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.194.212:43861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/krb5kdc/kudu_127.0.194.193.keytab
--principal=kudu/127.0.194.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.194.193:40383
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.194.193
--webserver_port=39049
--tserver_master_addrs=127.0.194.254:41965
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.194.193
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:48:23.105099 17413 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:48:23.106413 17413 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:48:23.120527 17461 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:48:23.120622 17458 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:48:23.120599 17459 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:48:23.121452 17413 server_base.cc:1034] running on GCE node
I20260926 03:48:23.122129 17413 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:48:23.123257 17413 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:48:23.124495 17413 hybrid_clock.cc:648] HybridClock initialized: now 1790394503124442 us; error 51 us; skew 500 ppm
I20260926 03:48:23.124867 17413 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:48:23 dist-test-slave-n2th krb5kdc[16260](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394503, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:48:23.128228 17413 init.cc:377] Logged in from keytab as kudu/127.0.194.193@KRBTEST.COM (short username kudu)
I20260926 03:48:23.129974 17413 webserver.cc:460] Webserver started at http://127.0.194.193:39049/ using document root <none> and password file <none>
I20260926 03:48:23.130704 17413 fs_manager.cc:359] Metadata directory not provided
I20260926 03:48:23.130923 17413 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:48:23.136531 17413 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 03:48:23.139501 17468 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:48:23.140844 17413 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 03:48:23.141023 17413 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/minicluster-data/ts-0/wal
uuid: "25395ab024974d1a88518d178c2e2ab2"
format_stamp: "Formatted at 2026-09-26 03:47:37 on dist-test-slave-n2th"
server_key: "512e244a8f11792c6844c3e2977049c0"
server_key_iv: "80df830967f0818d4ef48c077dd0b700"
server_key_version: "encryptionkey@0"
I20260926 03:48:23.142025 17413 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:48:23.162868 17413 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:48:23.167549 17413 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:48:23.167833 17413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:48:23.168815 17413 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:48:23.170581 17475 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
W20260926 03:48:23.176070 17475 ts_tablet_manager.cc:1852] T 78d97cb2f75840b596a4a0166c3b67a0 P 25395ab024974d1a88518d178c2e2ab2: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260926 03:48:23.177107 17475 ts_tablet_manager.cc:1905] T 78d97cb2f75840b596a4a0166c3b67a0 P 25395ab024974d1a88518d178c2e2ab2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:48:23.184583 17475 ts_tablet_manager.cc:1918] T 78d97cb2f75840b596a4a0166c3b67a0 P 25395ab024974d1a88518d178c2e2ab2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:48:23.184919 17475 ts_tablet_manager.cc:1939] T 78d97cb2f75840b596a4a0166c3b67a0 P 25395ab024974d1a88518d178c2e2ab2: Deleting consensus metadata
I20260926 03:48:23.185384 17413 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260926 03:48:23.185487 17413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.015s user 0.001s sys 0.000s
I20260926 03:48:23.185660 17413 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:48:23.185750 17413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:48:23.220129 17413 rpc_server.cc:307] RPC server started. Bound to: 127.0.194.193:40383
I20260926 03:48:23.220181 17581 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.194.193:40383 every 8 connection(s)
I20260926 03:48:23.221441 17413 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/minicluster-data/ts-0/data/info.pb
I20260926 03:48:23.223878 779 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu as pid 17413
I20260926 03:48:23.224532 779 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
/tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.194.194:41349
--local_ip_for_outbound_sockets=127.0.194.194
--tserver_master_addrs=127.0.194.254:41965
--webserver_port=37899
--webserver_interface=127.0.194.194
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/krb5kdc/kudu_127.0.194.194.keytab
--principal=kudu/127.0.194.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.194.212:43861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 03:48:23.228662 17584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:23.224447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:41965 (local address 127.0.194.193:54181)
0926 03:48:23.227394 (+ 2947us) negotiation.cc:107] Waiting for socket to connect
0926 03:48:23.227706 (+ 312us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.254:41965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2625,"thread_start_us":2427,"threads_started":1}
W20260926 03:48:23.229303 17582 heartbeater.cc:644] Failed to heartbeat to 127.0.194.254:41965 (0 consecutive failures): Network error: Failed to ping master at 127.0.194.254:41965: Client connection negotiation failed: client connection to 127.0.194.254:41965: connect: Connection refused (error 111)
I20260926 03:48:23.230249 17584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:23.229931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:41965 (local address 127.0.194.193:43979)
0926 03:48:23.230084 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0926 03:48:23.230151 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.254:41965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260926 03:48:23.231292 17584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:23.230837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:41965 (local address 127.0.194.193:53863)
0926 03:48:23.231008 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0926 03:48:23.231111 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.254:41965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260926 03:48:23.231585 17582 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 03:48:23.558362 17585 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:48:23.558658 17585 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:48:23.558832 17585 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:48:23.566107 17585 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:48:23.566229 17585 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:48:23.566344 17585 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.194.194
I20260926 03:48:23.576496 17585 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.194.212:43861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/krb5kdc/kudu_127.0.194.194.keytab
--principal=kudu/127.0.194.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.194.194:41349
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.194.194
--webserver_port=37899
--tserver_master_addrs=127.0.194.254:41965
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.194.194
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:48:23.577781 17585 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:48:23.579077 17585 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:48:23.592629 17593 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:48:23.592759 17591 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:48:23.592767 17590 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:48:23.593660 17585 server_base.cc:1034] running on GCE node
I20260926 03:48:23.594316 17585 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:48:23.595357 17585 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:48:23.596593 17585 hybrid_clock.cc:648] HybridClock initialized: now 1790394503596542 us; error 50 us; skew 500 ppm
I20260926 03:48:23.596935 17585 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:48:23 dist-test-slave-n2th krb5kdc[16260](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394503, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:48:23.600142 17585 init.cc:377] Logged in from keytab as kudu/127.0.194.194@KRBTEST.COM (short username kudu)
I20260926 03:48:23.601698 17585 webserver.cc:460] Webserver started at http://127.0.194.194:37899/ using document root <none> and password file <none>
I20260926 03:48:23.602409 17585 fs_manager.cc:359] Metadata directory not provided
I20260926 03:48:23.602557 17585 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:48:23.608150 17585 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260926 03:48:23.610929 17600 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:48:23.612347 17585 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 03:48:23.612527 17585 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/minicluster-data/ts-1/wal
uuid: "b5327dcf8cbc4adb804842b27d7ab913"
format_stamp: "Formatted at 2026-09-26 03:47:37 on dist-test-slave-n2th"
server_key: "6001f474c3d8de06bca8de24bdb8b31d"
server_key_iv: "308b140ceb06480085467a7e42793253"
server_key_version: "encryptionkey@0"
I20260926 03:48:23.613511 17585 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:48:23.629858 17585 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:48:23.633714 17585 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:48:23.634055 17585 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:48:23.635070 17585 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:48:23.636390 17585 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:48:23.636509 17585 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:48:23.636668 17585 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:48:23.636756 17585 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:48:23.670645 17585 rpc_server.cc:307] RPC server started. Bound to: 127.0.194.194:41349
I20260926 03:48:23.670748 17712 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.194.194:41349 every 8 connection(s)
I20260926 03:48:23.672140 17585 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/minicluster-data/ts-1/data/info.pb
I20260926 03:48:23.677024 17715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:23.675211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:41965 (local address 127.0.194.194:57341)
0926 03:48:23.675927 (+ 716us) negotiation.cc:107] Waiting for socket to connect
0926 03:48:23.676163 (+ 236us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.254:41965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":269,"threads_started":1}
W20260926 03:48:23.677771 17713 heartbeater.cc:644] Failed to heartbeat to 127.0.194.254:41965 (0 consecutive failures): Network error: Failed to ping master at 127.0.194.254:41965: Client connection negotiation failed: client connection to 127.0.194.254:41965: connect: Connection refused (error 111)
I20260926 03:48:23.678597 17715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:23.678255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:41965 (local address 127.0.194.194:56887)
0926 03:48:23.678426 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0926 03:48:23.678492 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.254:41965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260926 03:48:23.679471 17715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:23.679144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:41965 (local address 127.0.194.194:57813)
0926 03:48:23.679282 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0926 03:48:23.679361 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.254:41965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260926 03:48:23.679531 779 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu as pid 17585
W20260926 03:48:23.679709 17713 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 03:48:23.680199 779 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
/tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.194.195:41479
--local_ip_for_outbound_sockets=127.0.194.195
--tserver_master_addrs=127.0.194.254:41965
--webserver_port=35835
--webserver_interface=127.0.194.195
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/krb5kdc/kudu_127.0.194.195.keytab
--principal=kudu/127.0.194.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.194.212:43861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:48:24.016574 17716 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:48:24.016887 17716 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:48:24.017086 17716 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:48:24.024883 17716 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:48:24.025012 17716 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:48:24.025132 17716 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.194.195
I20260926 03:48:24.035315 17716 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.194.212:43861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/krb5kdc/kudu_127.0.194.195.keytab
--principal=kudu/127.0.194.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.194.195:41479
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.194.195
--webserver_port=35835
--tserver_master_addrs=127.0.194.254:41965
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.194.195
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:48:24.036598 17716 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:48:24.037863 17716 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:48:24.051265 17721 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:48:24.051326 17724 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:48:24.051352 17722 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:48:24.052170 17716 server_base.cc:1034] running on GCE node
I20260926 03:48:24.052765 17716 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:48:24.053697 17716 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:48:24.054883 17716 hybrid_clock.cc:648] HybridClock initialized: now 1790394504054836 us; error 36 us; skew 500 ppm
I20260926 03:48:24.055222 17716 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:48:24 dist-test-slave-n2th krb5kdc[16260](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394504, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:48:24.058472 17716 init.cc:377] Logged in from keytab as kudu/127.0.194.195@KRBTEST.COM (short username kudu)
I20260926 03:48:24.059962 17716 webserver.cc:460] Webserver started at http://127.0.194.195:35835/ using document root <none> and password file <none>
I20260926 03:48:24.060642 17716 fs_manager.cc:359] Metadata directory not provided
I20260926 03:48:24.060742 17716 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:48:24.066130 17716 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.002s
I20260926 03:48:24.068962 17731 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:48:24.070429 17716 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 03:48:24.070614 17716 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/minicluster-data/ts-2/wal
uuid: "e4ed0f05ec9949a5a78562546d5566d2"
format_stamp: "Formatted at 2026-09-26 03:47:38 on dist-test-slave-n2th"
server_key: "2e953a21f430b59eee8dcee318951152"
server_key_iv: "5324ade121b9e7de9d7c29bf0d51e50e"
server_key_version: "encryptionkey@0"
I20260926 03:48:24.071671 17716 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:48:24.087436 17716 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:48:24.091952 17716 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:48:24.092248 17716 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:48:24.093245 17716 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:48:24.094604 17716 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:48:24.094702 17716 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:48:24.094890 17716 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:48:24.094988 17716 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:48:24.129187 17716 rpc_server.cc:307] RPC server started. Bound to: 127.0.194.195:41479
I20260926 03:48:24.129240 17843 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.194.195:41479 every 8 connection(s)
I20260926 03:48:24.130666 17716 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790394221564982-779-0/minicluster-data/ts-2/data/info.pb
I20260926 03:48:24.135035 779 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu as pid 17716
I20260926 03:48:24.136185 17846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:24.133818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:41965 (local address 127.0.194.195:33927)
0926 03:48:24.134603 (+ 785us) negotiation.cc:107] Waiting for socket to connect
0926 03:48:24.134945 (+ 342us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.254:41965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":276,"threads_started":1}
W20260926 03:48:24.137017 17844 heartbeater.cc:644] Failed to heartbeat to 127.0.194.254:41965 (0 consecutive failures): Network error: Failed to ping master at 127.0.194.254:41965: Client connection negotiation failed: client connection to 127.0.194.254:41965: connect: Connection refused (error 111)
I20260926 03:48:24.137936 17846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:24.137605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:41965 (local address 127.0.194.195:43629)
0926 03:48:24.137741 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0926 03:48:24.137807 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.254:41965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260926 03:48:24.138800 17846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:24.138453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:41965 (local address 127.0.194.195:45685)
0926 03:48:24.138568 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0926 03:48:24.138638 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.254:41965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260926 03:48:24.139143 17844 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 03:48:24.232997 17848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:24.232204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:41965 (local address 127.0.194.193:55227)
0926 03:48:24.232678 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0926 03:48:24.232763 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.254:41965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":233,"threads_started":1}
I20260926 03:48:24.681133 17849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:24.680308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:41965 (local address 127.0.194.194:43139)
0926 03:48:24.680794 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0926 03:48:24.680906 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.254:41965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":227,"threads_started":1}
I20260926 03:48:25.140466 17850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:25.139672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:41965 (local address 127.0.194.195:46187)
0926 03:48:25.140133 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0926 03:48:25.140236 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.254:41965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":226,"threads_started":1}
I20260926 03:48:25.235021 17851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:25.234089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:41965 (local address 127.0.194.193:39361)
0926 03:48:25.234575 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0926 03:48:25.234670 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.254:41965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":249,"threads_started":1}
I20260926 03:48:25.682940 17852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:25.682143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:41965 (local address 127.0.194.194:41985)
0926 03:48:25.682612 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0926 03:48:25.682703 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.254:41965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":226,"threads_started":1}
I20260926 03:48:26.142791 17853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:26.141412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:41965 (local address 127.0.194.195:54123)
0926 03:48:26.142008 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0926 03:48:26.142144 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.254:41965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":327,"threads_started":1}
I20260926 03:48:26.236802 17854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:26.236016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:41965 (local address 127.0.194.193:34881)
0926 03:48:26.236548 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0926 03:48:26.236643 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.254:41965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":283,"threads_started":1}
I20260926 03:48:26.686477 17855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:26.683945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:41965 (local address 127.0.194.194:55825)
0926 03:48:26.686123 (+ 2178us) negotiation.cc:107] Waiting for socket to connect
0926 03:48:26.686237 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.254:41965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2045,"thread_start_us":1881,"threads_started":1}
I20260926 03:48:27.144511 17856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:27.143776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:41965 (local address 127.0.194.195:35001)
0926 03:48:27.144231 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0926 03:48:27.144331 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.254:41965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":232,"threads_started":1}
I20260926 03:48:27.238608 17857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:27.237803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:41965 (local address 127.0.194.193:42245)
0926 03:48:27.238318 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0926 03:48:27.238427 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.254:41965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":268,"threads_started":1}
I20260926 03:48:27.688266 17858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:27.687531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:41965 (local address 127.0.194.194:33333)
0926 03:48:27.687995 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0926 03:48:27.688081 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.254:41965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":239,"threads_started":1}
I20260926 03:48:28.146122 17859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:28.145356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:41965 (local address 127.0.194.195:43935)
0926 03:48:28.145835 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0926 03:48:28.145925 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.254:41965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":204,"threads_started":1}
I20260926 03:48:28.240365 17860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:28.239592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:41965 (local address 127.0.194.193:34631)
0926 03:48:28.240090 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0926 03:48:28.240191 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.254:41965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":262,"threads_started":1}
I20260926 03:48:28.690128 17861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:28.689371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:41965 (local address 127.0.194.194:57457)
0926 03:48:28.689868 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0926 03:48:28.689962 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.254:41965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":274,"threads_started":1}
I20260926 03:48:29.147974 17862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:29.147211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:41965 (local address 127.0.194.195:40577)
0926 03:48:29.147697 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0926 03:48:29.147784 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.254:41965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":251,"threads_started":1}
I20260926 03:48:29.242118 17863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:29.241394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:41965 (local address 127.0.194.193:53965)
0926 03:48:29.241830 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0926 03:48:29.241938 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.254:41965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":224,"threads_started":1}
I20260926 03:48:29.691881 17864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:29.691129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:41965 (local address 127.0.194.194:35301)
0926 03:48:29.691612 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0926 03:48:29.691708 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.254:41965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":249,"threads_started":1}
I20260926 03:48:30.149667 17865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:30.148928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:41965 (local address 127.0.194.195:58255)
0926 03:48:30.149405 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0926 03:48:30.149502 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.254:41965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":231,"threads_started":1}
I20260926 03:48:30.244129 17866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:30.243200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:41965 (local address 127.0.194.193:40323)
0926 03:48:30.243830 (+ 630us) negotiation.cc:107] Waiting for socket to connect
0926 03:48:30.243927 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.254:41965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":330,"threads_started":1}
I20260926 03:48:30.693787 17867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:30.692921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:41965 (local address 127.0.194.194:49617)
0926 03:48:30.693467 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0926 03:48:30.693560 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.254:41965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":222,"threads_started":1}
I20260926 03:48:31.151578 17868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:31.150685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:41965 (local address 127.0.194.195:54163)
0926 03:48:31.151260 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0926 03:48:31.151351 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.254:41965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":209,"threads_started":1}
I20260926 03:48:31.245963 17869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:31.245104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:41965 (local address 127.0.194.193:45679)
0926 03:48:31.245648 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0926 03:48:31.245755 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.254:41965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":252,"threads_started":1}
I20260926 03:48:31.695544 17870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:31.694816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:41965 (local address 127.0.194.194:36277)
0926 03:48:31.695281 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0926 03:48:31.695371 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.254:41965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":228,"threads_started":1}
I20260926 03:48:32.153455 17871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:32.152603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:41965 (local address 127.0.194.195:43347)
0926 03:48:32.153130 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0926 03:48:32.153224 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.254:41965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":250,"threads_started":1}
I20260926 03:48:32.247668 17872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:32.246952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:41965 (local address 127.0.194.193:35127)
0926 03:48:32.247420 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0926 03:48:32.247511 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.254:41965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":226,"threads_started":1}
I20260926 03:48:32.697216 17873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:32.696417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:41965 (local address 127.0.194.194:43045)
0926 03:48:32.696931 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0926 03:48:32.697038 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.254:41965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":261,"threads_started":1}
I20260926 03:48:33.155351 17874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:33.154475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:41965 (local address 127.0.194.195:51493)
0926 03:48:33.155060 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0926 03:48:33.155142 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.254:41965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":318,"threads_started":1}
I20260926 03:48:33.249287 17875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:33.248575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:41965 (local address 127.0.194.193:33579)
0926 03:48:33.249026 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0926 03:48:33.249109 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.254:41965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":252,"threads_started":1}
I20260926 03:48:33.698951 17876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:33.698158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:41965 (local address 127.0.194.194:40057)
0926 03:48:33.698604 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0926 03:48:33.698690 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.254:41965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":243,"threads_started":1}
I20260926 03:48:34.157186 17877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:34.156358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:41965 (local address 127.0.194.195:52229)
0926 03:48:34.156851 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0926 03:48:34.156974 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.254:41965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":287,"threads_started":1}
I20260926 03:48:34.251067 17878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:34.250210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:41965 (local address 127.0.194.193:51443)
0926 03:48:34.250789 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0926 03:48:34.250874 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.254:41965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":318,"threads_started":1}
I20260926 03:48:34.700821 17879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:34.699984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:41965 (local address 127.0.194.194:55947)
0926 03:48:34.700518 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0926 03:48:34.700628 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.254:41965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":220,"threads_started":1}
I20260926 03:48:35.159169 17880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:35.158341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:41965 (local address 127.0.194.195:55039)
0926 03:48:35.158875 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0926 03:48:35.158985 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.254:41965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":225,"threads_started":1}
I20260926 03:48:35.252765 17881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:35.251961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:41965 (local address 127.0.194.193:45721)
0926 03:48:35.252400 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0926 03:48:35.252507 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.254:41965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":259,"threads_started":1}
I20260926 03:48:35.702628 17882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:35.701889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:41965 (local address 127.0.194.194:49223)
0926 03:48:35.702347 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0926 03:48:35.702464 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.254:41965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":238,"threads_started":1}
I20260926 03:48:36.160925 17883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:36.160197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:41965 (local address 127.0.194.195:59559)
0926 03:48:36.160673 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0926 03:48:36.160763 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.254:41965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":255,"threads_started":1}
I20260926 03:48:36.254658 17884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:36.253876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:41965 (local address 127.0.194.193:54775)
0926 03:48:36.254369 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0926 03:48:36.254482 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.254:41965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":250,"threads_started":1}
I20260926 03:48:36.704530 17885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:36.703718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:41965 (local address 127.0.194.194:44227)
0926 03:48:36.704212 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0926 03:48:36.704335 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.254:41965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":249,"threads_started":1}
I20260926 03:48:37.162709 17886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:37.161940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:41965 (local address 127.0.194.195:34547)
0926 03:48:37.162417 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0926 03:48:37.162514 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.254:41965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":233,"threads_started":1}
I20260926 03:48:37.256412 17887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:37.255627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:41965 (local address 127.0.194.193:60275)
0926 03:48:37.256118 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0926 03:48:37.256217 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.254:41965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":245,"threads_started":1}
I20260926 03:48:37.706179 17888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:37.705472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:41965 (local address 127.0.194.194:50273)
0926 03:48:37.705909 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0926 03:48:37.705998 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.254:41965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":224,"threads_started":1}
I20260926 03:48:38.164665 17889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:38.163850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:41965 (local address 127.0.194.195:37145)
0926 03:48:38.164388 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0926 03:48:38.164480 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.254:41965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":297,"threads_started":1}
I20260926 03:48:38.258306 17890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:38.257569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:41965 (local address 127.0.194.193:41999)
0926 03:48:38.258025 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0926 03:48:38.258126 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.254:41965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":231,"threads_started":1}
I20260926 03:48:38.708069 17891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:38.707313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:41965 (local address 127.0.194.194:46459)
0926 03:48:38.707795 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0926 03:48:38.707888 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.254:41965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":226,"threads_started":1}
I20260926 03:48:39.146296 779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu pid 17413
I20260926 03:48:39.165859 17892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:39.147821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48552 (local address 127.0.194.193:40383)
0926 03:48:39.148532 (+ 711us) server_negotiation.cc:206] Beginning negotiation
0926 03:48:39.148555 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0926 03:48:39.148617 (+ 62us) server_negotiation.cc:407] Connection header received
0926 03:48:39.150528 (+ 1911us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:48:39.150539 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:48:39.151062 (+ 523us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:48:39.151370 (+ 308us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:48:39.152727 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:48:39.153981 (+ 1254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:48:39.155300 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:48:39.155984 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:48:39.156123 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:48:39.159404 (+ 3281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:48:39.159412 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:48:39.159429 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:48:39.159482 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:48:39.162309 (+ 2827us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:48:39.163151 (+ 842us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:48:39.163159 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:48:39.163166 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:48:39.163358 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:48:39.163794 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:48:39.163798 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:48:39.163800 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:48:39.164370 (+ 570us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:48:39.164617 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0926 03:48:39.165012 (+ 395us) server_negotiation.cc:299] Negotiation successful
0926 03:48:39.165528 (+ 516us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":475,"thread_start_us":282,"threads_started":1}
I20260926 03:48:39.166849 17893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:39.165824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:41965 (local address 127.0.194.195:38887)
0926 03:48:39.166510 (+ 686us) negotiation.cc:107] Waiting for socket to connect
0926 03:48:39.166601 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.254:41965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":423,"threads_started":1}
I20260926 03:48:39.167418 17577 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:48:39.253608 779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu with pid 17413
I20260926 03:48:39.271245 779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu pid 17585
I20260926 03:48:39.289954 17895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:39.272245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57914 (local address 127.0.194.194:41349)
0926 03:48:39.272772 (+ 527us) server_negotiation.cc:206] Beginning negotiation
0926 03:48:39.272787 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 03:48:39.272823 (+ 36us) server_negotiation.cc:407] Connection header received
0926 03:48:39.274388 (+ 1565us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:48:39.274398 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:48:39.274929 (+ 531us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:48:39.275300 (+ 371us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:48:39.276873 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:48:39.278359 (+ 1486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:48:39.279738 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:48:39.280445 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:48:39.280672 (+ 227us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:48:39.283447 (+ 2775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:48:39.283455 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:48:39.283472 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:48:39.283513 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:48:39.286306 (+ 2793us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:48:39.287241 (+ 935us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:48:39.287252 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:48:39.287266 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:48:39.287456 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:48:39.287860 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:48:39.287865 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:48:39.287872 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:48:39.288425 (+ 553us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:48:39.288741 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0926 03:48:39.289208 (+ 467us) server_negotiation.cc:299] Negotiation successful
0926 03:48:39.289661 (+ 453us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":383,"thread_start_us":240,"threads_started":1}
I20260926 03:48:39.291555 17708 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:48:39.376554 779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu with pid 17585
I20260926 03:48:39.393757 779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu pid 17716
I20260926 03:48:39.412499 17897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:39.394916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55112 (local address 127.0.194.195:41479)
0926 03:48:39.395556 (+ 640us) server_negotiation.cc:206] Beginning negotiation
0926 03:48:39.395570 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 03:48:39.395611 (+ 41us) server_negotiation.cc:407] Connection header received
0926 03:48:39.397221 (+ 1610us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:48:39.397232 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:48:39.397679 (+ 447us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:48:39.397969 (+ 290us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:48:39.399461 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:48:39.400720 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:48:39.401972 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:48:39.402667 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:48:39.402923 (+ 256us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:48:39.405744 (+ 2821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:48:39.405753 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:48:39.405770 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:48:39.405810 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:48:39.408869 (+ 3059us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:48:39.409655 (+ 786us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:48:39.409663 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:48:39.409670 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:48:39.409847 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:48:39.410260 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:48:39.410264 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:48:39.410268 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:48:39.410950 (+ 682us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:48:39.411263 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0926 03:48:39.411723 (+ 460us) server_negotiation.cc:299] Negotiation successful
0926 03:48:39.412173 (+ 450us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":465,"thread_start_us":307,"threads_started":1}
I20260926 03:48:39.413908 17839 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:48:39.498288 779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu with pid 17716
2026-09-26T03:48:39Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (64788 ms)
[----------] 30 tests from SecurityITest (297896 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790394221564982-779-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:48:39 dist-test-slave-n2th krb5kdc[17901](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:48:39 dist-test-slave-n2th krb5kdc[17901](info): set up 2 sockets
Sep 26 03:48:39 dist-test-slave-n2th krb5kdc[17901](info): commencing operation
krb5kdc: starting...
W20260926 03:48:40.607048 779 mini_kdc.cc:121] Time spent starting KDC: real 1.068s user 0.001s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790394221564982-779-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790394221564982-779-0/krb5kdc/test-user.keytab.
Sep 26 03:48:40 dist-test-slave-n2th krb5kdc[17901](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394520, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:48:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:48:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.194.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.194.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.194.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790394221564982-779-0/krb5kdc/kudu_127.0.194.254.keytab.
Entry for principal kudu/127.0.194.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790394221564982-779-0/krb5kdc/kudu_127.0.194.254.keytab.
I20260926 03:48:40.877979 779 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
/tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790394221564982-779-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790394221564982-779-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790394221564982-779-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790394221564982-779-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.194.254:43871
--webserver_interface=127.0.194.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790394221564982-779-0/krb5kdc/kudu_127.0.194.254.keytab
--principal=kudu/127.0.194.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.194.212:41843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.194.254:43871
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790394221564982-779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790394221564982-779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790394221564982-779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:48:41.211882 17914 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:48:41.212268 17914 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:48:41.212435 17914 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:48:41.219781 17914 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:48:41.219955 17914 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:48:41.220069 17914 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:48:41.220176 17914 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:48:41.220263 17914 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:48:41.230427 17914 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.194.212:41843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790394221564982-779-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790394221564982-779-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.194.254:43871
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790394221564982-779-0/krb5kdc/kudu_127.0.194.254.keytab
--principal=kudu/127.0.194.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.194.254:43871
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790394221564982-779-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.194.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790394221564982-779-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:48:41.231858 17914 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:48:41.233192 17914 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:48:41.245613 17922 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:48:41.245645 17920 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:48:41.245718 17919 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:48:41.246541 17914 server_base.cc:1034] running on GCE node
I20260926 03:48:41.247470 17914 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:48:41.249047 17914 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:48:41.250316 17914 hybrid_clock.cc:648] HybridClock initialized: now 1790394521250278 us; error 56 us; skew 500 ppm
I20260926 03:48:41.250702 17914 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:48:41 dist-test-slave-n2th krb5kdc[17901](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394521, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:48:41.254066 17914 init.cc:377] Logged in from keytab as kudu/127.0.194.254@KRBTEST.COM (short username kudu)
I20260926 03:48:41.255793 17914 webserver.cc:460] Webserver started at http://127.0.194.254:34303/ using document root <none> and password file <none>
I20260926 03:48:41.256521 17914 fs_manager.cc:359] Metadata directory not provided
I20260926 03:48:41.256670 17914 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:48:41.257006 17914 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:48:41.260134 17914 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790394221564982-779-0/minicluster-data/master-0/data/instance:
uuid: "3d47454a920a4df2af00289751b21f7b"
format_stamp: "Formatted at 2026-09-26 03:48:41 on dist-test-slave-n2th"
server_key: "c6ad722c1758c190c9a30d16e106252e"
server_key_iv: "951f437329600c96869e01fdaa24640c"
server_key_version: "encryptionkey@0"
I20260926 03:48:41.260946 17914 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790394221564982-779-0/minicluster-data/master-0/wal/instance:
uuid: "3d47454a920a4df2af00289751b21f7b"
format_stamp: "Formatted at 2026-09-26 03:48:41 on dist-test-slave-n2th"
server_key: "c6ad722c1758c190c9a30d16e106252e"
server_key_iv: "951f437329600c96869e01fdaa24640c"
server_key_version: "encryptionkey@0"
I20260926 03:48:41.266456 17914 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 03:48:41.270176 17929 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:48:41.271580 17914 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 03:48:41.271792 17914 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790394221564982-779-0/minicluster-data/master-0/data,/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790394221564982-779-0/minicluster-data/master-0/wal
uuid: "3d47454a920a4df2af00289751b21f7b"
format_stamp: "Formatted at 2026-09-26 03:48:41 on dist-test-slave-n2th"
server_key: "c6ad722c1758c190c9a30d16e106252e"
server_key_iv: "951f437329600c96869e01fdaa24640c"
server_key_version: "encryptionkey@0"
I20260926 03:48:41.272013 17914 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790394221564982-779-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790394221564982-779-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790394221564982-779-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:48:41.292636 17914 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:48:41.296875 17914 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:48:41.297180 17914 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:48:41.316988 17914 rpc_server.cc:307] RPC server started. Bound to: 127.0.194.254:43871
I20260926 03:48:41.317021 17980 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.194.254:43871 every 8 connection(s)
I20260926 03:48:41.318529 17914 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790394221564982-779-0/minicluster-data/master-0/data/info.pb
I20260926 03:48:41.323130 779 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu as pid 17914
I20260926 03:48:41.323244 17981 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:48:41.323380 779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790394221564982-779-0/minicluster-data/master-0/wal/instance
I20260926 03:48:41.324189 779 external_mini_cluster.cc:1428] Setting key ec8758063d72ebbae389273ccb2c0f04
I20260926 03:48:41.339277 17981 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d47454a920a4df2af00289751b21f7b: Bootstrap starting.
Sep 26 03:48:41 dist-test-slave-n2th krb5kdc[17901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394520, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.194.254@KRBTEST.COM
I20260926 03:48:41.342852 17981 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3d47454a920a4df2af00289751b21f7b: Neither blocks nor log segments found. Creating new log.
I20260926 03:48:41.344000 17981 log.cc:824] T 00000000000000000000000000000000 P 3d47454a920a4df2af00289751b21f7b: Log is configured to *not* fsync() on all Append() calls
I20260926 03:48:41.347034 17981 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d47454a920a4df2af00289751b21f7b: No bootstrap required, opened a new log
I20260926 03:48:41.348613 17984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:41.326405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38512 (local address 127.0.194.254:43871)
0926 03:48:41.327342 (+ 937us) server_negotiation.cc:206] Beginning negotiation
0926 03:48:41.327354 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 03:48:41.327398 (+ 44us) server_negotiation.cc:407] Connection header received
0926 03:48:41.328951 (+ 1553us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:48:41.328974 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:48:41.329401 (+ 427us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:48:41.329760 (+ 359us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:48:41.331741 (+ 1981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:48:41.333053 (+ 1312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:48:41.335028 (+ 1975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:48:41.335784 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:48:41.335941 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:48:41.341091 (+ 5150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:48:41.341099 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:48:41.341115 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:48:41.341155 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:48:41.344439 (+ 3284us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:48:41.345213 (+ 774us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:48:41.345220 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:48:41.345226 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:48:41.345401 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:48:41.345801 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:48:41.345805 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:48:41.345808 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:48:41.346339 (+ 531us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:48:41.346645 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0926 03:48:41.347188 (+ 543us) server_negotiation.cc:299] Negotiation successful
0926 03:48:41.347634 (+ 446us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":689,"thread_start_us":455,"threads_started":1}
I20260926 03:48:41.355474 17981 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3d47454a920a4df2af00289751b21f7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d47454a920a4df2af00289751b21f7b" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 43871 } }
I20260926 03:48:41.355872 17981 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3d47454a920a4df2af00289751b21f7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:48:41.355996 17981 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3d47454a920a4df2af00289751b21f7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d47454a920a4df2af00289751b21f7b, State: Initialized, Role: FOLLOWER
I20260926 03:48:41.356557 17981 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3d47454a920a4df2af00289751b21f7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d47454a920a4df2af00289751b21f7b" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 43871 } }
I20260926 03:48:41.356812 17981 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3d47454a920a4df2af00289751b21f7b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:48:41.356966 17981 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3d47454a920a4df2af00289751b21f7b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:48:41.357141 17981 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3d47454a920a4df2af00289751b21f7b [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:48:41.359551 17981 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3d47454a920a4df2af00289751b21f7b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d47454a920a4df2af00289751b21f7b" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 43871 } }
I20260926 03:48:41.360134 17981 leader_election.cc:302] T 00000000000000000000000000000000 P 3d47454a920a4df2af00289751b21f7b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3d47454a920a4df2af00289751b21f7b; no voters:
I20260926 03:48:41.360919 17981 leader_election.cc:288] T 00000000000000000000000000000000 P 3d47454a920a4df2af00289751b21f7b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:48:41.361068 17986 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3d47454a920a4df2af00289751b21f7b [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:48:41.361753 17986 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3d47454a920a4df2af00289751b21f7b [term 1 LEADER]: Becoming Leader. State: Replica: 3d47454a920a4df2af00289751b21f7b, State: Running, Role: LEADER
I20260926 03:48:41.362393 17986 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3d47454a920a4df2af00289751b21f7b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d47454a920a4df2af00289751b21f7b" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 43871 } }
I20260926 03:48:41.363415 17981 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3d47454a920a4df2af00289751b21f7b [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:48:41.365428 17987 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d47454a920a4df2af00289751b21f7b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3d47454a920a4df2af00289751b21f7b. Latest consensus state: current_term: 1 leader_uuid: "3d47454a920a4df2af00289751b21f7b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d47454a920a4df2af00289751b21f7b" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 43871 } } }
I20260926 03:48:41.365839 17987 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d47454a920a4df2af00289751b21f7b [sys.catalog]: This master's current role is: LEADER
I20260926 03:48:41.366562 17988 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d47454a920a4df2af00289751b21f7b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3d47454a920a4df2af00289751b21f7b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d47454a920a4df2af00289751b21f7b" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 43871 } } }
I20260926 03:48:41.367002 17988 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d47454a920a4df2af00289751b21f7b [sys.catalog]: This master's current role is: LEADER
I20260926 03:48:41.368310 17994 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:48:41.372637 17994 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:48:41.380970 17994 catalog_manager.cc:1347] Generated new cluster ID: 2073dfb364e14257832cb0758911ad58
I20260926 03:48:41.381093 17994 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:48:41.410777 17994 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:48:41.412158 17994 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:48:41.424469 17994 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3d47454a920a4df2af00289751b21f7b: Generated new TSK 0
I20260926 03:48:41.425401 17994 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.194.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.194.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.194.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790394221564982-779-0/krb5kdc/kudu_127.0.194.193.keytab.
Entry for principal kudu/127.0.194.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790394221564982-779-0/krb5kdc/kudu_127.0.194.193.keytab.
I20260926 03:48:41.490079 779 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
/tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790394221564982-779-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790394221564982-779-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790394221564982-779-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790394221564982-779-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.194.193:0
--local_ip_for_outbound_sockets=127.0.194.193
--webserver_interface=127.0.194.193
--webserver_port=0
--tserver_master_addrs=127.0.194.254:43871
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790394221564982-779-0/krb5kdc/kudu_127.0.194.193.keytab
--principal=kudu/127.0.194.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.194.212:41843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790394221564982-779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790394221564982-779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790394221564982-779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:48:41.842885 18007 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:48:41.843182 18007 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:48:41.843343 18007 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:48:41.851239 18007 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:48:41.851363 18007 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:48:41.851574 18007 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.194.193
I20260926 03:48:41.861949 18007 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.194.212:41843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790394221564982-779-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790394221564982-779-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790394221564982-779-0/krb5kdc/kudu_127.0.194.193.keytab
--principal=kudu/127.0.194.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.194.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790394221564982-779-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.194.193
--webserver_port=0
--tserver_master_addrs=127.0.194.254:43871
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.194.193
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790394221564982-779-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:48:41.863444 18007 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:48:41.864976 18007 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:48:41.879746 18015 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:48:41.879714 18013 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:48:41.879979 18012 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:48:41.880677 18007 server_base.cc:1034] running on GCE node
I20260926 03:48:41.881297 18007 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:48:41.882236 18007 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:48:41.883445 18007 hybrid_clock.cc:648] HybridClock initialized: now 1790394521883393 us; error 38 us; skew 500 ppm
I20260926 03:48:41.883777 18007 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:48:41 dist-test-slave-n2th krb5kdc[17901](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394521, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:48:41.895480 18007 init.cc:377] Logged in from keytab as kudu/127.0.194.193@KRBTEST.COM (short username kudu)
I20260926 03:48:41.897225 18007 webserver.cc:460] Webserver started at http://127.0.194.193:45245/ using document root <none> and password file <none>
I20260926 03:48:41.898142 18007 fs_manager.cc:359] Metadata directory not provided
I20260926 03:48:41.898350 18007 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:48:41.898705 18007 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:48:41.901906 18007 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790394221564982-779-0/minicluster-data/ts-0/data/instance:
uuid: "b24c437c50724421a3e2dba9f6251332"
format_stamp: "Formatted at 2026-09-26 03:48:41 on dist-test-slave-n2th"
server_key: "3d8172ebd431033c98b76b4e29112d44"
server_key_iv: "31e522f2c848422c2361e68ac0c62cc9"
server_key_version: "encryptionkey@0"
I20260926 03:48:41.902832 18007 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790394221564982-779-0/minicluster-data/ts-0/wal/instance:
uuid: "b24c437c50724421a3e2dba9f6251332"
format_stamp: "Formatted at 2026-09-26 03:48:41 on dist-test-slave-n2th"
server_key: "3d8172ebd431033c98b76b4e29112d44"
server_key_iv: "31e522f2c848422c2361e68ac0c62cc9"
server_key_version: "encryptionkey@0"
I20260926 03:48:41.908330 18007 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260926 03:48:41.912146 18022 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:48:41.913586 18007 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 03:48:41.913777 18007 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790394221564982-779-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790394221564982-779-0/minicluster-data/ts-0/wal
uuid: "b24c437c50724421a3e2dba9f6251332"
format_stamp: "Formatted at 2026-09-26 03:48:41 on dist-test-slave-n2th"
server_key: "3d8172ebd431033c98b76b4e29112d44"
server_key_iv: "31e522f2c848422c2361e68ac0c62cc9"
server_key_version: "encryptionkey@0"
I20260926 03:48:41.914016 18007 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790394221564982-779-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790394221564982-779-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790394221564982-779-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:48:41.945721 18007 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:48:41.950034 18007 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:48:41.950342 18007 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:48:41.951391 18007 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:48:41.952721 18007 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:48:41.952827 18007 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:48:41.952983 18007 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:48:41.953061 18007 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:48:41.986639 18007 rpc_server.cc:307] RPC server started. Bound to: 127.0.194.193:35125
I20260926 03:48:41.986686 18134 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.194.193:35125 every 8 connection(s)
I20260926 03:48:41.988050 18007 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790394221564982-779-0/minicluster-data/ts-0/data/info.pb
I20260926 03:48:41.989208 779 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu as pid 18007
I20260926 03:48:41.989418 779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790394221564982-779-0/minicluster-data/ts-0/wal/instance
I20260926 03:48:41.990259 779 external_mini_cluster.cc:1428] Setting key 17ab58c1fe1b2916b29d4164033b076e
Sep 26 03:48:42 dist-test-slave-n2th krb5kdc[17901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394521, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.193@KRBTEST.COM for kudu/127.0.194.254@KRBTEST.COM
I20260926 03:48:42.014801 18137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:41.991004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.193:50413 (local address 127.0.194.254:43871)
0926 03:48:41.991472 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0926 03:48:41.991478 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:48:41.992586 (+ 1108us) server_negotiation.cc:407] Connection header received
0926 03:48:41.997833 (+ 5247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:48:41.997838 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:48:41.997926 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:48:41.998028 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:48:42.000386 (+ 2358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:48:42.001109 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:48:42.002922 (+ 1813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:48:42.003680 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:48:42.003831 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:48:42.007514 (+ 3683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:48:42.007524 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:48:42.007532 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:48:42.007581 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:48:42.011265 (+ 3684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:48:42.012089 (+ 824us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:48:42.012097 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:48:42.012104 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:48:42.012265 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:48:42.012714 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:48:42.012722 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:48:42.012728 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:48:42.013132 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:48:42.013305 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0926 03:48:42.014016 (+ 711us) server_negotiation.cc:299] Negotiation successful
0926 03:48:42.014290 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":243,"threads_started":1}
I20260926 03:48:42.015654 18138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:41.991589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:43871 (local address 127.0.194.193:50413)
0926 03:48:41.992390 (+ 801us) negotiation.cc:107] Waiting for socket to connect
0926 03:48:41.992462 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0926 03:48:41.997454 (+ 4992us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:48:41.998353 (+ 899us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:48:41.998369 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:48:41.999283 (+ 914us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:48:42.000170 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:48:42.000199 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:48:42.001291 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:48:42.001299 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:48:42.002703 (+ 1404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:48:42.002716 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:48:42.003902 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:48:42.003906 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:48:42.004126 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:48:42.004847 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:48:42.004869 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:48:42.007234 (+ 2365us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:48:42.011431 (+ 4197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:48:42.011440 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:48:42.011456 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:48:42.011920 (+ 464us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:48:42.012381 (+ 461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:48:42.012384 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:48:42.012386 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:48:42.012591 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:48:42.013254 (+ 663us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:48:42.013260 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:48:42.013614 (+ 354us) client_negotiation.cc:769] Sending connection context
0926 03:48:42.013849 (+ 235us) client_negotiation.cc:240] Negotiation successful
0926 03:48:42.014375 (+ 526us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":306,"threads_started":1}
I20260926 03:48:42.017325 18135 heartbeater.cc:347] Connected to a master server at 127.0.194.254:43871
I20260926 03:48:42.017653 18135 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.194.194@KRBTEST.COM; defaulting to no policy
I20260926 03:48:42.018540 18135 heartbeater.cc:511] Master 127.0.194.254:43871 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.194.194@KRBTEST.COM" created.
I20260926 03:48:42.021155 17946 ts_manager.cc:194] Registered new tserver with Master: b24c437c50724421a3e2dba9f6251332 (127.0.194.193:35125)
I20260926 03:48:42.023384 17946 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.194.193@KRBTEST.COM'} at 127.0.194.193:50413
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.194.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790394221564982-779-0/krb5kdc/kudu_127.0.194.194.keytab.
Entry for principal kudu/127.0.194.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790394221564982-779-0/krb5kdc/kudu_127.0.194.194.keytab.
I20260926 03:48:42.052217 779 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
/tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790394221564982-779-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790394221564982-779-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790394221564982-779-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790394221564982-779-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.194.194:0
--local_ip_for_outbound_sockets=127.0.194.194
--webserver_interface=127.0.194.194
--webserver_port=0
--tserver_master_addrs=127.0.194.254:43871
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790394221564982-779-0/krb5kdc/kudu_127.0.194.194.keytab
--principal=kudu/127.0.194.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.194.212:41843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790394221564982-779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790394221564982-779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790394221564982-779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:48:42.391443 18141 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:48:42.391746 18141 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:48:42.391938 18141 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:48:42.399405 18141 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:48:42.399530 18141 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:48:42.399724 18141 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.194.194
I20260926 03:48:42.409721 18141 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.194.212:41843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790394221564982-779-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790394221564982-779-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790394221564982-779-0/krb5kdc/kudu_127.0.194.194.keytab
--principal=kudu/127.0.194.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.194.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790394221564982-779-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.194.194
--webserver_port=0
--tserver_master_addrs=127.0.194.254:43871
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.194.194
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790394221564982-779-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:48:42.411228 18141 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:48:42.412698 18141 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:48:42.426384 18146 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:48:42.426555 18149 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:48:42.426517 18147 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:48:42.427585 18141 server_base.cc:1034] running on GCE node
I20260926 03:48:42.428262 18141 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:48:42.429306 18141 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:48:42.430552 18141 hybrid_clock.cc:648] HybridClock initialized: now 1790394522430499 us; error 51 us; skew 500 ppm
I20260926 03:48:42.430961 18141 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:48:42 dist-test-slave-n2th krb5kdc[17901](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394522, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:48:42.434170 18141 init.cc:377] Logged in from keytab as kudu/127.0.194.194@KRBTEST.COM (short username kudu)
I20260926 03:48:42.435784 18141 webserver.cc:460] Webserver started at http://127.0.194.194:41937/ using document root <none> and password file <none>
I20260926 03:48:42.436517 18141 fs_manager.cc:359] Metadata directory not provided
I20260926 03:48:42.436666 18141 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:48:42.436998 18141 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:48:42.440127 18141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790394221564982-779-0/minicluster-data/ts-1/data/instance:
uuid: "164abcd754974ca7a9f92976c4c96da0"
format_stamp: "Formatted at 2026-09-26 03:48:42 on dist-test-slave-n2th"
server_key: "67f3f23fab6ac8619fd4f187eefab147"
server_key_iv: "ed65f56c0a9c16b0cbd08f6d4ad61704"
server_key_version: "encryptionkey@0"
I20260926 03:48:42.440975 18141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790394221564982-779-0/minicluster-data/ts-1/wal/instance:
uuid: "164abcd754974ca7a9f92976c4c96da0"
format_stamp: "Formatted at 2026-09-26 03:48:42 on dist-test-slave-n2th"
server_key: "67f3f23fab6ac8619fd4f187eefab147"
server_key_iv: "ed65f56c0a9c16b0cbd08f6d4ad61704"
server_key_version: "encryptionkey@0"
I20260926 03:48:42.446525 18141 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260926 03:48:42.450263 18156 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:48:42.451757 18141 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 03:48:42.451963 18141 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790394221564982-779-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790394221564982-779-0/minicluster-data/ts-1/wal
uuid: "164abcd754974ca7a9f92976c4c96da0"
format_stamp: "Formatted at 2026-09-26 03:48:42 on dist-test-slave-n2th"
server_key: "67f3f23fab6ac8619fd4f187eefab147"
server_key_iv: "ed65f56c0a9c16b0cbd08f6d4ad61704"
server_key_version: "encryptionkey@0"
I20260926 03:48:42.452184 18141 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790394221564982-779-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790394221564982-779-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790394221564982-779-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:48:42.474443 18141 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:48:42.479086 18141 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:48:42.479387 18141 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:48:42.480408 18141 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:48:42.481807 18141 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:48:42.481921 18141 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:48:42.482070 18141 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:48:42.482141 18141 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:48:42.516196 18141 rpc_server.cc:307] RPC server started. Bound to: 127.0.194.194:39659
I20260926 03:48:42.516356 18268 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.194.194:39659 every 8 connection(s)
I20260926 03:48:42.517554 18141 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790394221564982-779-0/minicluster-data/ts-1/data/info.pb
I20260926 03:48:42.518452 779 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu as pid 18141
I20260926 03:48:42.518659 779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790394221564982-779-0/minicluster-data/ts-1/wal/instance
I20260926 03:48:42.519235 779 external_mini_cluster.cc:1428] Setting key 4dd9d8158140e24bb5fedbadc4d09b6d
Sep 26 03:48:42 dist-test-slave-n2th krb5kdc[17901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394522, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.194@KRBTEST.COM for kudu/127.0.194.254@KRBTEST.COM
I20260926 03:48:42.538647 18271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:42.520568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.194:40109 (local address 127.0.194.254:43871)
0926 03:48:42.521052 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0926 03:48:42.521057 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:48:42.522198 (+ 1141us) server_negotiation.cc:407] Connection header received
0926 03:48:42.524135 (+ 1937us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:48:42.524141 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:48:42.524228 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:48:42.524357 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:48:42.525956 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:48:42.526689 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:48:42.528291 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:48:42.528756 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:48:42.528906 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:48:42.532754 (+ 3848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:48:42.532759 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:48:42.532762 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:48:42.532796 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:48:42.535509 (+ 2713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:48:42.536262 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:48:42.536268 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:48:42.536273 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:48:42.536406 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:48:42.536838 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:48:42.536844 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:48:42.536847 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:48:42.537122 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:48:42.537301 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0926 03:48:42.538023 (+ 722us) server_negotiation.cc:299] Negotiation successful
0926 03:48:42.538264 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":364,"thread_start_us":247,"threads_started":1}
I20260926 03:48:42.539687 18272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:42.521277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:43871 (local address 127.0.194.194:40109)
0926 03:48:42.522048 (+ 771us) negotiation.cc:107] Waiting for socket to connect
0926 03:48:42.522086 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 03:48:42.523863 (+ 1777us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:48:42.524609 (+ 746us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:48:42.524618 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:48:42.525202 (+ 584us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:48:42.525781 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:48:42.525802 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:48:42.526892 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:48:42.526897 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:48:42.528140 (+ 1243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:48:42.528152 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:48:42.529004 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:48:42.529012 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:48:42.529254 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:48:42.529984 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:48:42.530007 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:48:42.532491 (+ 2484us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:48:42.535657 (+ 3166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:48:42.535667 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:48:42.535687 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:48:42.536133 (+ 446us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:48:42.536521 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:48:42.536526 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:48:42.536530 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:48:42.536721 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:48:42.537254 (+ 533us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:48:42.537265 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:48:42.537668 (+ 403us) client_negotiation.cc:769] Sending connection context
0926 03:48:42.537948 (+ 280us) client_negotiation.cc:240] Negotiation successful
0926 03:48:42.538515 (+ 567us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":322,"threads_started":1}
I20260926 03:48:42.541260 18269 heartbeater.cc:347] Connected to a master server at 127.0.194.254:43871
I20260926 03:48:42.541637 18269 heartbeater.cc:464] Registering TS with master...
I20260926 03:48:42.542589 18269 heartbeater.cc:511] Master 127.0.194.254:43871 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.194.195@KRBTEST.COM; defaulting to no policy
I20260926 03:48:42.544456 17946 ts_manager.cc:194] Registered new tserver with Master: 164abcd754974ca7a9f92976c4c96da0 (127.0.194.194:39659)
I20260926 03:48:42.545403 17946 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.194.194@KRBTEST.COM'} at 127.0.194.194:40109
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.194.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.194.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790394221564982-779-0/krb5kdc/kudu_127.0.194.195.keytab.
Entry for principal kudu/127.0.194.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790394221564982-779-0/krb5kdc/kudu_127.0.194.195.keytab.
I20260926 03:48:42.575819 779 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
/tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790394221564982-779-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790394221564982-779-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790394221564982-779-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790394221564982-779-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.194.195:0
--local_ip_for_outbound_sockets=127.0.194.195
--webserver_interface=127.0.194.195
--webserver_port=0
--tserver_master_addrs=127.0.194.254:43871
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790394221564982-779-0/krb5kdc/kudu_127.0.194.195.keytab
--principal=kudu/127.0.194.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.194.212:41843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790394221564982-779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790394221564982-779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790394221564982-779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:48:42.916705 18275 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:48:42.917023 18275 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:48:42.917189 18275 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:48:42.924959 18275 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:48:42.925087 18275 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:48:42.925267 18275 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.194.195
I20260926 03:48:42.935379 18275 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.194.212:41843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790394221564982-779-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790394221564982-779-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790394221564982-779-0/krb5kdc/kudu_127.0.194.195.keytab
--principal=kudu/127.0.194.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.194.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790394221564982-779-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.194.195
--webserver_port=0
--tserver_master_addrs=127.0.194.254:43871
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.194.195
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790394221564982-779-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:48:42.936743 18275 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:48:42.938163 18275 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:48:42.951825 18280 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:48:42.951843 18281 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:48:42.952062 18283 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:48:42.952644 18275 server_base.cc:1034] running on GCE node
I20260926 03:48:42.953459 18275 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:48:42.954432 18275 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:48:42.955657 18275 hybrid_clock.cc:648] HybridClock initialized: now 1790394522955575 us; error 73 us; skew 500 ppm
I20260926 03:48:42.956058 18275 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:48:42 dist-test-slave-n2th krb5kdc[17901](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394522, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:48:42.959609 18275 init.cc:377] Logged in from keytab as kudu/127.0.194.195@KRBTEST.COM (short username kudu)
I20260926 03:48:42.961174 18275 webserver.cc:460] Webserver started at http://127.0.194.195:46723/ using document root <none> and password file <none>
I20260926 03:48:42.961905 18275 fs_manager.cc:359] Metadata directory not provided
I20260926 03:48:42.962036 18275 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:48:42.962364 18275 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:48:42.965592 18275 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790394221564982-779-0/minicluster-data/ts-2/data/instance:
uuid: "563b7da0894c4257af73b4ef2f17ba62"
format_stamp: "Formatted at 2026-09-26 03:48:42 on dist-test-slave-n2th"
server_key: "96db31eb8b4f97733035f412209138c7"
server_key_iv: "f27b19ba5c8b81ae3eff5947326e524b"
server_key_version: "encryptionkey@0"
I20260926 03:48:42.966428 18275 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790394221564982-779-0/minicluster-data/ts-2/wal/instance:
uuid: "563b7da0894c4257af73b4ef2f17ba62"
format_stamp: "Formatted at 2026-09-26 03:48:42 on dist-test-slave-n2th"
server_key: "96db31eb8b4f97733035f412209138c7"
server_key_iv: "f27b19ba5c8b81ae3eff5947326e524b"
server_key_version: "encryptionkey@0"
I20260926 03:48:42.972105 18275 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260926 03:48:42.975872 18290 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:48:42.977515 18275 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 03:48:42.977730 18275 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790394221564982-779-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790394221564982-779-0/minicluster-data/ts-2/wal
uuid: "563b7da0894c4257af73b4ef2f17ba62"
format_stamp: "Formatted at 2026-09-26 03:48:42 on dist-test-slave-n2th"
server_key: "96db31eb8b4f97733035f412209138c7"
server_key_iv: "f27b19ba5c8b81ae3eff5947326e524b"
server_key_version: "encryptionkey@0"
I20260926 03:48:42.977973 18275 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790394221564982-779-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790394221564982-779-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790394221564982-779-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:48:43.002776 18275 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:48:43.007243 18275 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:48:43.007570 18275 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:48:43.008585 18275 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:48:43.009980 18275 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:48:43.010058 18275 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:48:43.010215 18275 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:48:43.010288 18275 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:48:43.026980 18135 heartbeater.cc:503] Master 127.0.194.254:43871 was elected leader, sending a full tablet report...
I20260926 03:48:43.044369 18275 rpc_server.cc:307] RPC server started. Bound to: 127.0.194.195:41765
I20260926 03:48:43.044463 18402 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.194.195:41765 every 8 connection(s)
I20260926 03:48:43.045756 18275 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790394221564982-779-0/minicluster-data/ts-2/data/info.pb
I20260926 03:48:43.051290 779 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu as pid 18275
I20260926 03:48:43.051481 779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790394221564982-779-0/minicluster-data/ts-2/wal/instance
I20260926 03:48:43.052052 779 external_mini_cluster.cc:1428] Setting key bcf11bc1a165bd591a1fde380abb12ed
Sep 26 03:48:43 dist-test-slave-n2th krb5kdc[17901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394522, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.195@KRBTEST.COM for kudu/127.0.194.254@KRBTEST.COM
I20260926 03:48:43.067905 18405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:43.048516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.195:60787 (local address 127.0.194.254:43871)
0926 03:48:43.049039 (+ 523us) server_negotiation.cc:206] Beginning negotiation
0926 03:48:43.049044 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:48:43.049872 (+ 828us) server_negotiation.cc:407] Connection header received
0926 03:48:43.052343 (+ 2471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:48:43.052347 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:48:43.052422 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:48:43.052584 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:48:43.054413 (+ 1829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:48:43.055386 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:48:43.057201 (+ 1815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:48:43.057635 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:48:43.057805 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:48:43.061338 (+ 3533us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:48:43.061343 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:48:43.061346 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:48:43.061381 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:48:43.064352 (+ 2971us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:48:43.065134 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:48:43.065138 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:48:43.065141 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:48:43.065263 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:48:43.065737 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:48:43.065740 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:48:43.065742 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:48:43.066071 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:48:43.066225 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0926 03:48:43.067190 (+ 965us) server_negotiation.cc:299] Negotiation successful
0926 03:48:43.067437 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":401,"thread_start_us":276,"threads_started":1}
I20260926 03:48:43.068907 18406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:43.048984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:43871 (local address 127.0.194.195:60787)
0926 03:48:43.049709 (+ 725us) negotiation.cc:107] Waiting for socket to connect
0926 03:48:43.049747 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 03:48:43.051988 (+ 2241us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:48:43.052865 (+ 877us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:48:43.052873 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:48:43.053553 (+ 680us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:48:43.054215 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:48:43.054243 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:48:43.055526 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:48:43.055536 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:48:43.057074 (+ 1538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:48:43.057087 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:48:43.057958 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:48:43.057966 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:48:43.058180 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:48:43.058894 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:48:43.058916 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:48:43.061105 (+ 2189us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:48:43.064491 (+ 3386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:48:43.064498 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:48:43.064513 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:48:43.065023 (+ 510us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:48:43.065361 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:48:43.065366 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:48:43.065369 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:48:43.065632 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:48:43.066197 (+ 565us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:48:43.066203 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:48:43.066665 (+ 462us) client_negotiation.cc:769] Sending connection context
0926 03:48:43.067099 (+ 434us) client_negotiation.cc:240] Negotiation successful
0926 03:48:43.067620 (+ 521us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":281,"threads_started":1}
I20260926 03:48:43.070338 18403 heartbeater.cc:347] Connected to a master server at 127.0.194.254:43871
I20260926 03:48:43.070647 18403 heartbeater.cc:464] Registering TS with master...
I20260926 03:48:43.071449 18403 heartbeater.cc:511] Master 127.0.194.254:43871 requested a full tablet report, sending...
I20260926 03:48:43.072922 17946 ts_manager.cc:194] Registered new tserver with Master: 563b7da0894c4257af73b4ef2f17ba62 (127.0.194.195:41765)
I20260926 03:48:43.073726 17946 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.194.195@KRBTEST.COM'} at 127.0.194.195:60787
I20260926 03:48:43.080653 779 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 03:48:43.097357 18405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:43.085209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38524 (local address 127.0.194.254:43871)
0926 03:48:43.085386 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0926 03:48:43.085390 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:48:43.085892 (+ 502us) server_negotiation.cc:407] Connection header received
0926 03:48:43.086047 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:48:43.086051 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:48:43.086114 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:48:43.086196 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:48:43.087772 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:48:43.088516 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:48:43.089789 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:48:43.090175 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:48:43.090266 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:48:43.092619 (+ 2353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:48:43.092624 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:48:43.092627 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:48:43.092658 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:48:43.095115 (+ 2457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:48:43.095758 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:48:43.095762 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:48:43.095764 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:48:43.095877 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:48:43.096199 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:48:43.096202 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:48:43.096204 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:48:43.096517 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:48:43.096697 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0926 03:48:43.096969 (+ 272us) server_negotiation.cc:299] Negotiation successful
0926 03:48:43.097134 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 03:48:43.101728 17946 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38524:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 03:48:43.108826 17946 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 03:48:43.132459 18419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:43.122690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53856 (local address 127.0.194.194:39659)
0926 03:48:43.123662 (+ 972us) server_negotiation.cc:206] Beginning negotiation
0926 03:48:43.123671 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:48:43.123694 (+ 23us) server_negotiation.cc:407] Connection header received
0926 03:48:43.123771 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:48:43.123777 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:48:43.123977 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:48:43.124141 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:48:43.125701 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:48:43.127416 (+ 1715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:48:43.130113 (+ 2697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:48:43.131140 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:48:43.131291 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:48:43.131493 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0926 03:48:43.131894 (+ 401us) server_negotiation.cc:299] Negotiation successful
0926 03:48:43.132077 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":826,"thread_start_us":710,"threads_started":1}
I20260926 03:48:43.133322 18417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:43.121628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38506 (local address 127.0.194.195:41765)
0926 03:48:43.122268 (+ 640us) server_negotiation.cc:206] Beginning negotiation
0926 03:48:43.122280 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 03:48:43.123265 (+ 985us) server_negotiation.cc:407] Connection header received
0926 03:48:43.123579 (+ 314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:48:43.123587 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:48:43.123841 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:48:43.124001 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:48:43.125107 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:48:43.126565 (+ 1458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:48:43.129438 (+ 2873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:48:43.131032 (+ 1594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:48:43.131165 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:48:43.131412 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0926 03:48:43.132848 (+ 1436us) server_negotiation.cc:299] Negotiation successful
0926 03:48:43.133047 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":451,"thread_start_us":312,"threads_started":1}
I20260926 03:48:43.133528 18421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:43.123819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51956 (local address 127.0.194.193:35125)
0926 03:48:43.124459 (+ 640us) server_negotiation.cc:206] Beginning negotiation
0926 03:48:43.124470 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 03:48:43.125225 (+ 755us) server_negotiation.cc:407] Connection header received
0926 03:48:43.125331 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:48:43.125340 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:48:43.125600 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:48:43.125776 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:48:43.126568 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:48:43.127756 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:48:43.130148 (+ 2392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:48:43.131645 (+ 1497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:48:43.131793 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:48:43.132024 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0926 03:48:43.133003 (+ 979us) server_negotiation.cc:299] Negotiation successful
0926 03:48:43.133189 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":447,"thread_start_us":316,"threads_started":1}
I20260926 03:48:43.133641 18416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:43.121439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.195:41765 (local address 127.0.0.1:38506)
0926 03:48:43.122751 (+ 1312us) negotiation.cc:107] Waiting for socket to connect
0926 03:48:43.123179 (+ 428us) client_negotiation.cc:174] Beginning negotiation
0926 03:48:43.123380 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:48:43.124357 (+ 977us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:48:43.124374 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:48:43.124411 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:48:43.124893 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:48:43.124905 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:48:43.126716 (+ 1811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:48:43.126724 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:48:43.129238 (+ 2514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:48:43.129253 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:48:43.131587 (+ 2334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:48:43.131595 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:48:43.131802 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:48:43.131833 (+ 31us) client_negotiation.cc:769] Sending connection context
0926 03:48:43.131922 (+ 89us) client_negotiation.cc:240] Negotiation successful
0926 03:48:43.132150 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1001,"mutex_wait_us":69,"spinlock_wait_cycles":3968,"thread_start_us":834,"threads_started":1}
I20260926 03:48:43.133788 18420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:43.123242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.193:35125 (local address 127.0.0.1:51956)
0926 03:48:43.124938 (+ 1696us) negotiation.cc:107] Waiting for socket to connect
0926 03:48:43.124960 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 03:48:43.125067 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:48:43.126038 (+ 971us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:48:43.126043 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:48:43.126073 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:48:43.126382 (+ 309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:48:43.126393 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:48:43.127910 (+ 1517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:48:43.127915 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:48:43.129980 (+ 2065us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:48:43.129993 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:48:43.132110 (+ 2117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:48:43.132119 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:48:43.132304 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:48:43.132333 (+ 29us) client_negotiation.cc:769] Sending connection context
0926 03:48:43.132426 (+ 93us) client_negotiation.cc:240] Negotiation successful
0926 03:48:43.132579 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1530,"mutex_wait_us":8,"thread_start_us":1349,"threads_started":1}
I20260926 03:48:43.133813 18418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:43.122169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.194:39659 (local address 127.0.0.1:53856)
0926 03:48:43.122752 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0926 03:48:43.122829 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0926 03:48:43.122997 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:48:43.124357 (+ 1360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:48:43.124366 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:48:43.124406 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:48:43.125496 (+ 1090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:48:43.125509 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:48:43.127560 (+ 2051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:48:43.127566 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:48:43.129922 (+ 2356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:48:43.129949 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:48:43.131354 (+ 1405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:48:43.131370 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:48:43.131601 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:48:43.131649 (+ 48us) client_negotiation.cc:769] Sending connection context
0926 03:48:43.132829 (+ 1180us) client_negotiation.cc:240] Negotiation successful
0926 03:48:43.132976 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":415,"mutex_wait_us":660,"spinlock_wait_cycles":768,"thread_start_us":725,"threads_started":1}
I20260926 03:48:43.137643 18204 tablet_service.cc:1467] Processing CreateTablet for tablet da52667d427e4cee874dbdfb7674913b (DEFAULT_TABLE table=test-table [id=4a2ab37d66cb48edabfb78376b9ae7c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:48:43.137661 18338 tablet_service.cc:1467] Processing CreateTablet for tablet da52667d427e4cee874dbdfb7674913b (DEFAULT_TABLE table=test-table [id=4a2ab37d66cb48edabfb78376b9ae7c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:48:43.139016 18204 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da52667d427e4cee874dbdfb7674913b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:48:43.139137 18338 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da52667d427e4cee874dbdfb7674913b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:48:43.139343 18070 tablet_service.cc:1467] Processing CreateTablet for tablet da52667d427e4cee874dbdfb7674913b (DEFAULT_TABLE table=test-table [id=4a2ab37d66cb48edabfb78376b9ae7c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:48:43.140723 18070 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da52667d427e4cee874dbdfb7674913b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:48:43.151757 18422 tablet_bootstrap.cc:492] T da52667d427e4cee874dbdfb7674913b P 164abcd754974ca7a9f92976c4c96da0: Bootstrap starting.
I20260926 03:48:43.154246 18423 tablet_bootstrap.cc:492] T da52667d427e4cee874dbdfb7674913b P 563b7da0894c4257af73b4ef2f17ba62: Bootstrap starting.
I20260926 03:48:43.157091 18422 tablet_bootstrap.cc:654] T da52667d427e4cee874dbdfb7674913b P 164abcd754974ca7a9f92976c4c96da0: Neither blocks nor log segments found. Creating new log.
I20260926 03:48:43.158532 18423 tablet_bootstrap.cc:654] T da52667d427e4cee874dbdfb7674913b P 563b7da0894c4257af73b4ef2f17ba62: Neither blocks nor log segments found. Creating new log.
I20260926 03:48:43.158587 18422 log.cc:824] T da52667d427e4cee874dbdfb7674913b P 164abcd754974ca7a9f92976c4c96da0: Log is configured to *not* fsync() on all Append() calls
I20260926 03:48:43.160030 18423 log.cc:824] T da52667d427e4cee874dbdfb7674913b P 563b7da0894c4257af73b4ef2f17ba62: Log is configured to *not* fsync() on all Append() calls
I20260926 03:48:43.161597 18424 tablet_bootstrap.cc:492] T da52667d427e4cee874dbdfb7674913b P b24c437c50724421a3e2dba9f6251332: Bootstrap starting.
I20260926 03:48:43.162361 18422 tablet_bootstrap.cc:492] T da52667d427e4cee874dbdfb7674913b P 164abcd754974ca7a9f92976c4c96da0: No bootstrap required, opened a new log
I20260926 03:48:43.162693 18422 ts_tablet_manager.cc:1397] T da52667d427e4cee874dbdfb7674913b P 164abcd754974ca7a9f92976c4c96da0: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260926 03:48:43.163606 18423 tablet_bootstrap.cc:492] T da52667d427e4cee874dbdfb7674913b P 563b7da0894c4257af73b4ef2f17ba62: No bootstrap required, opened a new log
I20260926 03:48:43.163968 18423 ts_tablet_manager.cc:1397] T da52667d427e4cee874dbdfb7674913b P 563b7da0894c4257af73b4ef2f17ba62: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.004s
I20260926 03:48:43.165129 18424 tablet_bootstrap.cc:654] T da52667d427e4cee874dbdfb7674913b P b24c437c50724421a3e2dba9f6251332: Neither blocks nor log segments found. Creating new log.
I20260926 03:48:43.166528 18424 log.cc:824] T da52667d427e4cee874dbdfb7674913b P b24c437c50724421a3e2dba9f6251332: Log is configured to *not* fsync() on all Append() calls
I20260926 03:48:43.174779 18422 raft_consensus.cc:348] T da52667d427e4cee874dbdfb7674913b P 164abcd754974ca7a9f92976c4c96da0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "563b7da0894c4257af73b4ef2f17ba62" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 41765 } } peers { permanent_uuid: "b24c437c50724421a3e2dba9f6251332" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 35125 } } peers { permanent_uuid: "164abcd754974ca7a9f92976c4c96da0" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 39659 } }
I20260926 03:48:43.175355 18422 raft_consensus.cc:374] T da52667d427e4cee874dbdfb7674913b P 164abcd754974ca7a9f92976c4c96da0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:48:43.175509 18422 raft_consensus.cc:729] T da52667d427e4cee874dbdfb7674913b P 164abcd754974ca7a9f92976c4c96da0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 164abcd754974ca7a9f92976c4c96da0, State: Initialized, Role: FOLLOWER
I20260926 03:48:43.175467 18423 raft_consensus.cc:348] T da52667d427e4cee874dbdfb7674913b P 563b7da0894c4257af73b4ef2f17ba62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "563b7da0894c4257af73b4ef2f17ba62" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 41765 } } peers { permanent_uuid: "b24c437c50724421a3e2dba9f6251332" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 35125 } } peers { permanent_uuid: "164abcd754974ca7a9f92976c4c96da0" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 39659 } }
I20260926 03:48:43.176050 18423 raft_consensus.cc:374] T da52667d427e4cee874dbdfb7674913b P 563b7da0894c4257af73b4ef2f17ba62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:48:43.176170 18423 raft_consensus.cc:729] T da52667d427e4cee874dbdfb7674913b P 563b7da0894c4257af73b4ef2f17ba62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 563b7da0894c4257af73b4ef2f17ba62, State: Initialized, Role: FOLLOWER
I20260926 03:48:43.176398 18422 consensus_queue.cc:260] T da52667d427e4cee874dbdfb7674913b P 164abcd754974ca7a9f92976c4c96da0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "563b7da0894c4257af73b4ef2f17ba62" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 41765 } } peers { permanent_uuid: "b24c437c50724421a3e2dba9f6251332" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 35125 } } peers { permanent_uuid: "164abcd754974ca7a9f92976c4c96da0" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 39659 } }
I20260926 03:48:43.176808 18423 consensus_queue.cc:260] T da52667d427e4cee874dbdfb7674913b P 563b7da0894c4257af73b4ef2f17ba62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "563b7da0894c4257af73b4ef2f17ba62" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 41765 } } peers { permanent_uuid: "b24c437c50724421a3e2dba9f6251332" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 35125 } } peers { permanent_uuid: "164abcd754974ca7a9f92976c4c96da0" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 39659 } }
I20260926 03:48:43.179239 18269 heartbeater.cc:503] Master 127.0.194.254:43871 was elected leader, sending a full tablet report...
I20260926 03:48:43.179250 18424 tablet_bootstrap.cc:492] T da52667d427e4cee874dbdfb7674913b P b24c437c50724421a3e2dba9f6251332: No bootstrap required, opened a new log
I20260926 03:48:43.179551 18424 ts_tablet_manager.cc:1397] T da52667d427e4cee874dbdfb7674913b P b24c437c50724421a3e2dba9f6251332: Time spent bootstrapping tablet: real 0.019s user 0.002s sys 0.015s
I20260926 03:48:43.179737 18403 heartbeater.cc:503] Master 127.0.194.254:43871 was elected leader, sending a full tablet report...
I20260926 03:48:43.180029 18422 ts_tablet_manager.cc:1428] T da52667d427e4cee874dbdfb7674913b P 164abcd754974ca7a9f92976c4c96da0: Time spent starting tablet: real 0.017s user 0.009s sys 0.006s
I20260926 03:48:43.181612 18423 ts_tablet_manager.cc:1428] T da52667d427e4cee874dbdfb7674913b P 563b7da0894c4257af73b4ef2f17ba62: Time spent starting tablet: real 0.018s user 0.012s sys 0.004s
I20260926 03:48:43.188978 18424 raft_consensus.cc:348] T da52667d427e4cee874dbdfb7674913b P b24c437c50724421a3e2dba9f6251332 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "563b7da0894c4257af73b4ef2f17ba62" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 41765 } } peers { permanent_uuid: "b24c437c50724421a3e2dba9f6251332" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 35125 } } peers { permanent_uuid: "164abcd754974ca7a9f92976c4c96da0" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 39659 } }
I20260926 03:48:43.189360 18424 raft_consensus.cc:374] T da52667d427e4cee874dbdfb7674913b P b24c437c50724421a3e2dba9f6251332 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:48:43.189435 18424 raft_consensus.cc:729] T da52667d427e4cee874dbdfb7674913b P b24c437c50724421a3e2dba9f6251332 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b24c437c50724421a3e2dba9f6251332, State: Initialized, Role: FOLLOWER
I20260926 03:48:43.190033 18424 consensus_queue.cc:260] T da52667d427e4cee874dbdfb7674913b P b24c437c50724421a3e2dba9f6251332 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "563b7da0894c4257af73b4ef2f17ba62" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 41765 } } peers { permanent_uuid: "b24c437c50724421a3e2dba9f6251332" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 35125 } } peers { permanent_uuid: "164abcd754974ca7a9f92976c4c96da0" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 39659 } }
I20260926 03:48:43.191962 18424 ts_tablet_manager.cc:1428] T da52667d427e4cee874dbdfb7674913b P b24c437c50724421a3e2dba9f6251332: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
W20260926 03:48:43.240520 18136 tablet.cc:2369] T da52667d427e4cee874dbdfb7674913b P b24c437c50724421a3e2dba9f6251332: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 03:48:43.269799 18270 tablet.cc:2369] T da52667d427e4cee874dbdfb7674913b P 164abcd754974ca7a9f92976c4c96da0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 03:48:43.288681 18429 raft_consensus.cc:482] T da52667d427e4cee874dbdfb7674913b P 563b7da0894c4257af73b4ef2f17ba62 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 03:48:43.288969 18429 raft_consensus.cc:504] T da52667d427e4cee874dbdfb7674913b P 563b7da0894c4257af73b4ef2f17ba62 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "563b7da0894c4257af73b4ef2f17ba62" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 41765 } } peers { permanent_uuid: "b24c437c50724421a3e2dba9f6251332" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 35125 } } peers { permanent_uuid: "164abcd754974ca7a9f92976c4c96da0" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 39659 } }
I20260926 03:48:43.290624 18429 leader_election.cc:288] T da52667d427e4cee874dbdfb7674913b P 563b7da0894c4257af73b4ef2f17ba62 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b24c437c50724421a3e2dba9f6251332 (127.0.194.193:35125), 164abcd754974ca7a9f92976c4c96da0 (127.0.194.194:39659)
I20260926 03:48:43.297112 18421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:43.291150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.195:42157 (local address 127.0.194.193:35125)
0926 03:48:43.291350 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0926 03:48:43.291357 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:48:43.291506 (+ 149us) server_negotiation.cc:407] Connection header received
0926 03:48:43.291745 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:48:43.291753 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:48:43.291833 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:48:43.291953 (+ 120us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:48:43.292430 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:48:43.293334 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:48:43.295269 (+ 1935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:48:43.296179 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:48:43.296302 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:48:43.296356 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 03:48:43.296722 (+ 366us) server_negotiation.cc:299] Negotiation successful
0926 03:48:43.296860 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 03:48:43.297132 18406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:43.291046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.193:35125 (local address 127.0.194.195:42157)
0926 03:48:43.291373 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0926 03:48:43.291398 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0926 03:48:43.291577 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:48:43.292003 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:48:43.292009 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:48:43.292041 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:48:43.292269 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:48:43.292284 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:48:43.293476 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:48:43.293482 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:48:43.295115 (+ 1633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:48:43.295131 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:48:43.296335 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:48:43.296341 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:48:43.296543 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:48:43.296582 (+ 39us) client_negotiation.cc:769] Sending connection context
0926 03:48:43.296678 (+ 96us) client_negotiation.cc:240] Negotiation successful
0926 03:48:43.296857 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"mutex_wait_us":96}
W20260926 03:48:43.297457 18404 tablet.cc:2369] T da52667d427e4cee874dbdfb7674913b P 563b7da0894c4257af73b4ef2f17ba62: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 03:48:43.297554 18419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:43.291162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.195:36821 (local address 127.0.194.194:39659)
0926 03:48:43.291350 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0926 03:48:43.291357 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:48:43.291995 (+ 638us) server_negotiation.cc:407] Connection header received
0926 03:48:43.292153 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:48:43.292158 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:48:43.292234 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:48:43.292334 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:48:43.292918 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:48:43.293721 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:48:43.295638 (+ 1917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:48:43.296543 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:48:43.296702 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:48:43.296782 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 03:48:43.297140 (+ 358us) server_negotiation.cc:299] Negotiation successful
0926 03:48:43.297287 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 03:48:43.298174 18090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da52667d427e4cee874dbdfb7674913b" candidate_uuid: "563b7da0894c4257af73b4ef2f17ba62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b24c437c50724421a3e2dba9f6251332" is_pre_election: true
I20260926 03:48:43.298350 18431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:43.291083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.194:39659 (local address 127.0.194.195:36821)
0926 03:48:43.291882 (+ 799us) negotiation.cc:107] Waiting for socket to connect
0926 03:48:43.291901 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 03:48:43.292020 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:48:43.292372 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:48:43.292379 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:48:43.292404 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:48:43.292771 (+ 367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:48:43.292783 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:48:43.293872 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:48:43.293876 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:48:43.295478 (+ 1602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:48:43.295491 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:48:43.296774 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:48:43.296779 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:48:43.296952 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:48:43.296990 (+ 38us) client_negotiation.cc:769] Sending connection context
0926 03:48:43.297083 (+ 93us) client_negotiation.cc:240] Negotiation successful
0926 03:48:43.297946 (+ 863us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":676,"mutex_wait_us":19,"thread_start_us":463,"threads_started":1}
I20260926 03:48:43.298808 18090 raft_consensus.cc:2454] T da52667d427e4cee874dbdfb7674913b P b24c437c50724421a3e2dba9f6251332 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 563b7da0894c4257af73b4ef2f17ba62 in term 0.
I20260926 03:48:43.299582 18292 leader_election.cc:302] T da52667d427e4cee874dbdfb7674913b P 563b7da0894c4257af73b4ef2f17ba62 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 563b7da0894c4257af73b4ef2f17ba62, b24c437c50724421a3e2dba9f6251332; no voters:
I20260926 03:48:43.299482 18224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da52667d427e4cee874dbdfb7674913b" candidate_uuid: "563b7da0894c4257af73b4ef2f17ba62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "164abcd754974ca7a9f92976c4c96da0" is_pre_election: true
I20260926 03:48:43.300010 18429 raft_consensus.cc:2781] T da52667d427e4cee874dbdfb7674913b P 563b7da0894c4257af73b4ef2f17ba62 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 03:48:43.300048 18224 raft_consensus.cc:2454] T da52667d427e4cee874dbdfb7674913b P 164abcd754974ca7a9f92976c4c96da0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 563b7da0894c4257af73b4ef2f17ba62 in term 0.
I20260926 03:48:43.300164 18429 raft_consensus.cc:482] T da52667d427e4cee874dbdfb7674913b P 563b7da0894c4257af73b4ef2f17ba62 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 03:48:43.300292 18429 raft_consensus.cc:3037] T da52667d427e4cee874dbdfb7674913b P 563b7da0894c4257af73b4ef2f17ba62 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:48:43.302603 18429 raft_consensus.cc:504] T da52667d427e4cee874dbdfb7674913b P 563b7da0894c4257af73b4ef2f17ba62 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "563b7da0894c4257af73b4ef2f17ba62" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 41765 } } peers { permanent_uuid: "b24c437c50724421a3e2dba9f6251332" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 35125 } } peers { permanent_uuid: "164abcd754974ca7a9f92976c4c96da0" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 39659 } }
I20260926 03:48:43.303303 18429 leader_election.cc:288] T da52667d427e4cee874dbdfb7674913b P 563b7da0894c4257af73b4ef2f17ba62 [CANDIDATE]: Term 1 election: Requested vote from peers b24c437c50724421a3e2dba9f6251332 (127.0.194.193:35125), 164abcd754974ca7a9f92976c4c96da0 (127.0.194.194:39659)
I20260926 03:48:43.303948 18224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da52667d427e4cee874dbdfb7674913b" candidate_uuid: "563b7da0894c4257af73b4ef2f17ba62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "164abcd754974ca7a9f92976c4c96da0"
I20260926 03:48:43.303964 18090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da52667d427e4cee874dbdfb7674913b" candidate_uuid: "563b7da0894c4257af73b4ef2f17ba62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b24c437c50724421a3e2dba9f6251332"
I20260926 03:48:43.304193 18090 raft_consensus.cc:3037] T da52667d427e4cee874dbdfb7674913b P b24c437c50724421a3e2dba9f6251332 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:48:43.304191 18224 raft_consensus.cc:3037] T da52667d427e4cee874dbdfb7674913b P 164abcd754974ca7a9f92976c4c96da0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:48:43.307657 18090 raft_consensus.cc:2454] T da52667d427e4cee874dbdfb7674913b P b24c437c50724421a3e2dba9f6251332 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 563b7da0894c4257af73b4ef2f17ba62 in term 1.
I20260926 03:48:43.307657 18224 raft_consensus.cc:2454] T da52667d427e4cee874dbdfb7674913b P 164abcd754974ca7a9f92976c4c96da0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 563b7da0894c4257af73b4ef2f17ba62 in term 1.
I20260926 03:48:43.308234 18292 leader_election.cc:302] T da52667d427e4cee874dbdfb7674913b P 563b7da0894c4257af73b4ef2f17ba62 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 563b7da0894c4257af73b4ef2f17ba62, b24c437c50724421a3e2dba9f6251332; no voters:
I20260926 03:48:43.308614 18429 raft_consensus.cc:2781] T da52667d427e4cee874dbdfb7674913b P 563b7da0894c4257af73b4ef2f17ba62 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:48:43.309069 18429 raft_consensus.cc:686] T da52667d427e4cee874dbdfb7674913b P 563b7da0894c4257af73b4ef2f17ba62 [term 1 LEADER]: Becoming Leader. State: Replica: 563b7da0894c4257af73b4ef2f17ba62, State: Running, Role: LEADER
I20260926 03:48:43.309643 18429 consensus_queue.cc:237] T da52667d427e4cee874dbdfb7674913b P 563b7da0894c4257af73b4ef2f17ba62 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "563b7da0894c4257af73b4ef2f17ba62" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 41765 } } peers { permanent_uuid: "b24c437c50724421a3e2dba9f6251332" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 35125 } } peers { permanent_uuid: "164abcd754974ca7a9f92976c4c96da0" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 39659 } }
I20260926 03:48:43.314284 17945 catalog_manager.cc:5495] T da52667d427e4cee874dbdfb7674913b P 563b7da0894c4257af73b4ef2f17ba62 reported cstate change: term changed from 0 to 1, leader changed from <none> to 563b7da0894c4257af73b4ef2f17ba62 (127.0.194.195). New cstate: current_term: 1 leader_uuid: "563b7da0894c4257af73b4ef2f17ba62" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "563b7da0894c4257af73b4ef2f17ba62" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 41765 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b24c437c50724421a3e2dba9f6251332" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 35125 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "164abcd754974ca7a9f92976c4c96da0" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 39659 } health_report { overall_health: UNKNOWN } } }
I20260926 03:48:43.332887 18417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:43.326893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38520 (local address 127.0.194.195:41765)
0926 03:48:43.327084 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0926 03:48:43.327088 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:48:43.327106 (+ 18us) server_negotiation.cc:407] Connection header received
0926 03:48:43.327192 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:48:43.327196 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:48:43.327282 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:48:43.327413 (+ 131us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:48:43.327836 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:48:43.328550 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:48:43.330619 (+ 2069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:48:43.331115 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:48:43.331213 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:48:43.331772 (+ 559us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:48:43.332311 (+ 539us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:48:43.332425 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0926 03:48:43.332553 (+ 128us) server_negotiation.cc:299] Negotiation successful
0926 03:48:43.332623 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 03:48:43.341290 18224 raft_consensus.cc:1261] T da52667d427e4cee874dbdfb7674913b P 164abcd754974ca7a9f92976c4c96da0 [term 1 FOLLOWER]: Refusing update from remote peer 563b7da0894c4257af73b4ef2f17ba62: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:48:43.341292 18090 raft_consensus.cc:1261] T da52667d427e4cee874dbdfb7674913b P b24c437c50724421a3e2dba9f6251332 [term 1 FOLLOWER]: Refusing update from remote peer 563b7da0894c4257af73b4ef2f17ba62: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:48:43.342375 18432 consensus_queue.cc:1035] T da52667d427e4cee874dbdfb7674913b P 563b7da0894c4257af73b4ef2f17ba62 [LEADER]: Connected to new peer: Peer: permanent_uuid: "164abcd754974ca7a9f92976c4c96da0" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 39659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:48:43.342765 18429 consensus_queue.cc:1035] T da52667d427e4cee874dbdfb7674913b P 563b7da0894c4257af73b4ef2f17ba62 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b24c437c50724421a3e2dba9f6251332" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 35125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:48:43.355227 18435 mvcc.cc:205] Tried to move back new op lower bound from 7333455967595622400 to 7333455967480180736. Current Snapshot: MvccSnapshot[applied={T|T < 7333455967595622400}]
I20260926 03:48:43.373126 17945 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38524:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:48:43.373435 17945 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38524:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:48:43.377453 17945 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3d47454a920a4df2af00289751b21f7b: Sending DeleteTablet for 3 replicas of tablet da52667d427e4cee874dbdfb7674913b
I20260926 03:48:43.378855 18204 tablet_service.cc:1514] Processing DeleteTablet for tablet da52667d427e4cee874dbdfb7674913b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:48:43 UTC) from {username='kudu', principal='kudu/127.0.194.254@KRBTEST.COM'} at 127.0.0.1:53856
I20260926 03:48:43.379128 18338 tablet_service.cc:1514] Processing DeleteTablet for tablet da52667d427e4cee874dbdfb7674913b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:48:43 UTC) from {username='kudu', principal='kudu/127.0.194.254@KRBTEST.COM'} at 127.0.0.1:38506
I20260926 03:48:43.379351 18070 tablet_service.cc:1514] Processing DeleteTablet for tablet da52667d427e4cee874dbdfb7674913b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:48:43 UTC) from {username='kudu', principal='kudu/127.0.194.254@KRBTEST.COM'} at 127.0.0.1:51956
I20260926 03:48:43.380327 779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu pid 18007
I20260926 03:48:43.391315 18445 tablet_replica.cc:330] T da52667d427e4cee874dbdfb7674913b P 164abcd754974ca7a9f92976c4c96da0: stopping tablet replica
I20260926 03:48:43.392153 18445 raft_consensus.cc:2229] T da52667d427e4cee874dbdfb7674913b P 164abcd754974ca7a9f92976c4c96da0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:48:43.393311 18445 raft_consensus.cc:2258] T da52667d427e4cee874dbdfb7674913b P 164abcd754974ca7a9f92976c4c96da0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:48:43.394326 18446 tablet_replica.cc:330] T da52667d427e4cee874dbdfb7674913b P 563b7da0894c4257af73b4ef2f17ba62: stopping tablet replica
I20260926 03:48:43.394920 18446 raft_consensus.cc:2229] T da52667d427e4cee874dbdfb7674913b P 563b7da0894c4257af73b4ef2f17ba62 [term 1 LEADER]: Raft consensus shutting down.
I20260926 03:48:43.395215 18447 tablet_replica.cc:330] T da52667d427e4cee874dbdfb7674913b P b24c437c50724421a3e2dba9f6251332: stopping tablet replica
I20260926 03:48:43.395797 18446 raft_consensus.cc:2258] T da52667d427e4cee874dbdfb7674913b P 563b7da0894c4257af73b4ef2f17ba62 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:48:43.396806 18445 ts_tablet_manager.cc:1905] T da52667d427e4cee874dbdfb7674913b P 164abcd754974ca7a9f92976c4c96da0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:48:43.396843 18447 raft_consensus.cc:2229] T da52667d427e4cee874dbdfb7674913b P b24c437c50724421a3e2dba9f6251332 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:48:43.397406 18447 raft_consensus.cc:2258] T da52667d427e4cee874dbdfb7674913b P b24c437c50724421a3e2dba9f6251332 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:48:43.399241 18446 ts_tablet_manager.cc:1905] T da52667d427e4cee874dbdfb7674913b P 563b7da0894c4257af73b4ef2f17ba62: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:48:43.399658 18447 ts_tablet_manager.cc:1905] T da52667d427e4cee874dbdfb7674913b P b24c437c50724421a3e2dba9f6251332: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:48:43.404335 18445 ts_tablet_manager.cc:1918] T da52667d427e4cee874dbdfb7674913b P 164abcd754974ca7a9f92976c4c96da0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:48:43.404613 18445 log.cc:1196] T da52667d427e4cee874dbdfb7674913b P 164abcd754974ca7a9f92976c4c96da0: Deleting WAL directory at /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790394221564982-779-0/minicluster-data/ts-1/wal/wals/da52667d427e4cee874dbdfb7674913b
I20260926 03:48:43.405272 18445 ts_tablet_manager.cc:1939] T da52667d427e4cee874dbdfb7674913b P 164abcd754974ca7a9f92976c4c96da0: Deleting consensus metadata
I20260926 03:48:43.406952 18447 ts_tablet_manager.cc:1918] T da52667d427e4cee874dbdfb7674913b P b24c437c50724421a3e2dba9f6251332: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:48:43.407055 18446 ts_tablet_manager.cc:1918] T da52667d427e4cee874dbdfb7674913b P 563b7da0894c4257af73b4ef2f17ba62: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:48:43.407186 18447 log.cc:1196] T da52667d427e4cee874dbdfb7674913b P b24c437c50724421a3e2dba9f6251332: Deleting WAL directory at /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790394221564982-779-0/minicluster-data/ts-0/wal/wals/da52667d427e4cee874dbdfb7674913b
I20260926 03:48:43.407514 17930 catalog_manager.cc:4841] TS 164abcd754974ca7a9f92976c4c96da0 (127.0.194.194:39659): tablet da52667d427e4cee874dbdfb7674913b (table test-table [id=4a2ab37d66cb48edabfb78376b9ae7c3]) successfully deleted
I20260926 03:48:43.407785 18447 ts_tablet_manager.cc:1939] T da52667d427e4cee874dbdfb7674913b P b24c437c50724421a3e2dba9f6251332: Deleting consensus metadata
I20260926 03:48:43.408960 18446 log.cc:1196] T da52667d427e4cee874dbdfb7674913b P 563b7da0894c4257af73b4ef2f17ba62: Deleting WAL directory at /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790394221564982-779-0/minicluster-data/ts-2/wal/wals/da52667d427e4cee874dbdfb7674913b
I20260926 03:48:43.409686 17931 catalog_manager.cc:4841] TS b24c437c50724421a3e2dba9f6251332 (127.0.194.193:35125): tablet da52667d427e4cee874dbdfb7674913b (table test-table [id=4a2ab37d66cb48edabfb78376b9ae7c3]) successfully deleted
I20260926 03:48:43.410336 18446 ts_tablet_manager.cc:1939] T da52667d427e4cee874dbdfb7674913b P 563b7da0894c4257af73b4ef2f17ba62: Deleting consensus metadata
I20260926 03:48:43.411883 17931 catalog_manager.cc:4841] TS 563b7da0894c4257af73b4ef2f17ba62 (127.0.194.195:41765): tablet da52667d427e4cee874dbdfb7674913b (table test-table [id=4a2ab37d66cb48edabfb78376b9ae7c3]) successfully deleted
Sep 26 03:48:43 dist-test-slave-n2th krb5kdc[17901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394520, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.194.193@KRBTEST.COM
I20260926 03:48:43.419783 18421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:43.382285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51964 (local address 127.0.194.193:35125)
0926 03:48:43.382525 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0926 03:48:43.382534 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:48:43.388031 (+ 5497us) server_negotiation.cc:407] Connection header received
0926 03:48:43.389319 (+ 1288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:48:43.389328 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:48:43.389457 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:48:43.389555 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:48:43.393919 (+ 4364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:48:43.395188 (+ 1269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:48:43.407303 (+ 12115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:48:43.408082 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:48:43.408265 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:48:43.413468 (+ 5203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:48:43.413483 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:48:43.413497 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:48:43.413535 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:48:43.416655 (+ 3120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:48:43.417455 (+ 800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:48:43.417463 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:48:43.417466 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:48:43.417658 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:48:43.418090 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:48:43.418095 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:48:43.418100 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:48:43.418468 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:48:43.418825 (+ 357us) server_negotiation.cc:1035] Waiting for connection context
0926 03:48:43.419176 (+ 351us) server_negotiation.cc:299] Negotiation successful
0926 03:48:43.419412 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260926 03:48:43.420468 18130 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:48:43.516175 779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu with pid 18007
I20260926 03:48:43.540596 779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu pid 18141
Sep 26 03:48:43 dist-test-slave-n2th krb5kdc[17901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394520, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.194.194@KRBTEST.COM
I20260926 03:48:43.558265 18419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:43.541392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53866 (local address 127.0.194.194:39659)
0926 03:48:43.541568 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0926 03:48:43.541572 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:48:43.541640 (+ 68us) server_negotiation.cc:407] Connection header received
0926 03:48:43.541816 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:48:43.541820 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:48:43.541882 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:48:43.541994 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:48:43.543726 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:48:43.544456 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:48:43.546264 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:48:43.546803 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:48:43.546926 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:48:43.552046 (+ 5120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:48:43.552061 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:48:43.552083 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:48:43.552119 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:48:43.555226 (+ 3107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:48:43.556057 (+ 831us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:48:43.556065 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:48:43.556067 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:48:43.556245 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:48:43.556651 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:48:43.556659 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:48:43.556661 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:48:43.557046 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:48:43.557351 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0926 03:48:43.557672 (+ 321us) server_negotiation.cc:299] Negotiation successful
0926 03:48:43.557892 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 03:48:43.559109 18264 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:48:43.649663 779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu with pid 18141
I20260926 03:48:43.667361 779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu pid 18275
Sep 26 03:48:43 dist-test-slave-n2th krb5kdc[17901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394520, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.194.195@KRBTEST.COM
I20260926 03:48:43.685253 18417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:43.668277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38532 (local address 127.0.194.195:41765)
0926 03:48:43.668476 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0926 03:48:43.668482 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:48:43.668504 (+ 22us) server_negotiation.cc:407] Connection header received
0926 03:48:43.668599 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:48:43.668606 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:48:43.668685 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:48:43.668797 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:48:43.670041 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:48:43.671333 (+ 1292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:48:43.672605 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:48:43.673304 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:48:43.673440 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:48:43.678073 (+ 4633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:48:43.678086 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:48:43.678107 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:48:43.678164 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:48:43.682118 (+ 3954us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:48:43.683021 (+ 903us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:48:43.683031 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:48:43.683035 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:48:43.683235 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:48:43.683619 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:48:43.683624 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:48:43.683628 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:48:43.684108 (+ 480us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:48:43.684437 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0926 03:48:43.684604 (+ 167us) server_negotiation.cc:299] Negotiation successful
0926 03:48:43.684829 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 03:48:43.686120 18398 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:48:43.814220 779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu with pid 18275
I20260926 03:48:43.833820 779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu pid 17914
I20260926 03:48:43.835005 17976 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:48:43.947137 779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu with pid 17914
2026-09-26T03:48:43Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4451 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-26T03:48:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:48:44Z Disabled control of system clock
I20260926 03:48:44.025173 779 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
/tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790394221564982-779-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790394221564982-779-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790394221564982-779-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790394221564982-779-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.194.254:34587
--webserver_interface=127.0.194.254
--webserver_port=0
--builtin_ntp_servers=127.0.194.212:37287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.194.254:34587
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 03:48:44.360034 18456 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:48:44.360330 18456 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:48:44.360410 18456 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:48:44.367889 18456 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:48:44.368040 18456 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:48:44.368119 18456 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:48:44.368204 18456 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:48:44.368251 18456 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:48:44.377451 18456 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.194.212:37287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790394221564982-779-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790394221564982-779-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.194.254:34587
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.194.254:34587
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790394221564982-779-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.194.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790394221564982-779-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:48:44.378854 18456 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:48:44.380599 18456 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:48:44.392870 18461 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:48:44.392911 18462 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:48:44.392819 18464 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:48:44.393728 18456 server_base.cc:1034] running on GCE node
I20260926 03:48:44.394555 18456 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:48:44.396056 18456 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:48:44.397285 18456 hybrid_clock.cc:648] HybridClock initialized: now 1790394524397230 us; error 55 us; skew 500 ppm
I20260926 03:48:44.397637 18456 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 03:48:44.400014 18456 webserver.cc:460] Webserver started at http://127.0.194.254:38907/ using document root <none> and password file <none>
I20260926 03:48:44.400735 18456 fs_manager.cc:359] Metadata directory not provided
I20260926 03:48:44.400851 18456 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:48:44.401183 18456 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:48:44.404211 18456 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790394221564982-779-0/minicluster-data/master-0/data/instance:
uuid: "6bdba152b50e4503a278636584aedcf0"
format_stamp: "Formatted at 2026-09-26 03:48:44 on dist-test-slave-n2th"
server_key: "64d0dc413011cf75ff287dae6b2b1295"
server_key_iv: "f4fbf720b6a87ffd74a62a4612dfed7c"
server_key_version: "encryptionkey@0"
I20260926 03:48:44.405031 18456 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790394221564982-779-0/minicluster-data/master-0/wal/instance:
uuid: "6bdba152b50e4503a278636584aedcf0"
format_stamp: "Formatted at 2026-09-26 03:48:44 on dist-test-slave-n2th"
server_key: "64d0dc413011cf75ff287dae6b2b1295"
server_key_iv: "f4fbf720b6a87ffd74a62a4612dfed7c"
server_key_version: "encryptionkey@0"
I20260926 03:48:44.410410 18456 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260926 03:48:44.413956 18470 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:48:44.415357 18456 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 03:48:44.415552 18456 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790394221564982-779-0/minicluster-data/master-0/data,/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790394221564982-779-0/minicluster-data/master-0/wal
uuid: "6bdba152b50e4503a278636584aedcf0"
format_stamp: "Formatted at 2026-09-26 03:48:44 on dist-test-slave-n2th"
server_key: "64d0dc413011cf75ff287dae6b2b1295"
server_key_iv: "f4fbf720b6a87ffd74a62a4612dfed7c"
server_key_version: "encryptionkey@0"
I20260926 03:48:44.415777 18456 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790394221564982-779-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790394221564982-779-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790394221564982-779-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:48:44.437115 18456 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:48:44.438011 18456 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:48:44.438311 18456 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:48:44.457059 18456 rpc_server.cc:307] RPC server started. Bound to: 127.0.194.254:34587
I20260926 03:48:44.457098 18521 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.194.254:34587 every 8 connection(s)
I20260926 03:48:44.458460 18456 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790394221564982-779-0/minicluster-data/master-0/data/info.pb
I20260926 03:48:44.460085 779 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu as pid 18456
I20260926 03:48:44.460273 779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790394221564982-779-0/minicluster-data/master-0/wal/instance
I20260926 03:48:44.460695 779 external_mini_cluster.cc:1428] Setting key 4efaf66b1a3be55fd5025784410138bf
I20260926 03:48:44.462673 18522 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:48:44.474977 18525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:44.462979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53386 (local address 127.0.194.254:34587)
0926 03:48:44.464029 (+ 1050us) server_negotiation.cc:206] Beginning negotiation
0926 03:48:44.464040 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 03:48:44.464079 (+ 39us) server_negotiation.cc:407] Connection header received
0926 03:48:44.465608 (+ 1529us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:48:44.465628 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:48:44.466097 (+ 469us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:48:44.466464 (+ 367us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:48:44.467117 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:48:44.468450 (+ 1333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:48:44.470435 (+ 1985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:48:44.471140 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:48:44.471267 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:48:44.472542 (+ 1275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:48:44.472550 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:48:44.472563 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:48:44.473135 (+ 572us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:48:44.473206 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:48:44.473232 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:48:44.473322 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 03:48:44.473560 (+ 238us) server_negotiation.cc:299] Negotiation successful
0926 03:48:44.473905 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":630,"thread_start_us":408,"threads_started":1}
I20260926 03:48:44.476238 18522 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6bdba152b50e4503a278636584aedcf0: Bootstrap starting.
I20260926 03:48:44.479548 18522 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6bdba152b50e4503a278636584aedcf0: Neither blocks nor log segments found. Creating new log.
I20260926 03:48:44.480548 18522 log.cc:824] T 00000000000000000000000000000000 P 6bdba152b50e4503a278636584aedcf0: Log is configured to *not* fsync() on all Append() calls
I20260926 03:48:44.483579 18522 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6bdba152b50e4503a278636584aedcf0: No bootstrap required, opened a new log
I20260926 03:48:44.491434 18522 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6bdba152b50e4503a278636584aedcf0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bdba152b50e4503a278636584aedcf0" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 34587 } }
I20260926 03:48:44.491843 18522 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6bdba152b50e4503a278636584aedcf0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:48:44.491959 18522 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6bdba152b50e4503a278636584aedcf0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bdba152b50e4503a278636584aedcf0, State: Initialized, Role: FOLLOWER
I20260926 03:48:44.492535 18522 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6bdba152b50e4503a278636584aedcf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bdba152b50e4503a278636584aedcf0" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 34587 } }
I20260926 03:48:44.492769 18522 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6bdba152b50e4503a278636584aedcf0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:48:44.492933 18522 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6bdba152b50e4503a278636584aedcf0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:48:44.493108 18522 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6bdba152b50e4503a278636584aedcf0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:48:44.495349 18522 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6bdba152b50e4503a278636584aedcf0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bdba152b50e4503a278636584aedcf0" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 34587 } }
I20260926 03:48:44.495851 18522 leader_election.cc:302] T 00000000000000000000000000000000 P 6bdba152b50e4503a278636584aedcf0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6bdba152b50e4503a278636584aedcf0; no voters:
I20260926 03:48:44.496491 18522 leader_election.cc:288] T 00000000000000000000000000000000 P 6bdba152b50e4503a278636584aedcf0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:48:44.496634 18527 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6bdba152b50e4503a278636584aedcf0 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:48:44.497720 18527 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6bdba152b50e4503a278636584aedcf0 [term 1 LEADER]: Becoming Leader. State: Replica: 6bdba152b50e4503a278636584aedcf0, State: Running, Role: LEADER
I20260926 03:48:44.498219 18527 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6bdba152b50e4503a278636584aedcf0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bdba152b50e4503a278636584aedcf0" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 34587 } }
I20260926 03:48:44.498644 18522 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6bdba152b50e4503a278636584aedcf0 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:48:44.501029 18528 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6bdba152b50e4503a278636584aedcf0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6bdba152b50e4503a278636584aedcf0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bdba152b50e4503a278636584aedcf0" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 34587 } } }
I20260926 03:48:44.501353 18528 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6bdba152b50e4503a278636584aedcf0 [sys.catalog]: This master's current role is: LEADER
I20260926 03:48:44.502198 18535 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:48:44.502074 18529 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6bdba152b50e4503a278636584aedcf0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6bdba152b50e4503a278636584aedcf0. Latest consensus state: current_term: 1 leader_uuid: "6bdba152b50e4503a278636584aedcf0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bdba152b50e4503a278636584aedcf0" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 34587 } } }
I20260926 03:48:44.502436 18529 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6bdba152b50e4503a278636584aedcf0 [sys.catalog]: This master's current role is: LEADER
I20260926 03:48:44.508261 18535 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:48:44.517601 18535 catalog_manager.cc:1347] Generated new cluster ID: f4ad93f0c12c4bcd9d63c677ebe15a25
I20260926 03:48:44.517736 18535 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:48:44.543268 18535 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:48:44.544605 18535 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:48:44.553155 18535 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6bdba152b50e4503a278636584aedcf0: Generated new TSK 0
I20260926 03:48:44.553990 18535 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 03:48:44.565372 779 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
/tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790394221564982-779-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790394221564982-779-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790394221564982-779-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790394221564982-779-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.194.193:0
--local_ip_for_outbound_sockets=127.0.194.193
--webserver_interface=127.0.194.193
--webserver_port=0
--tserver_master_addrs=127.0.194.254:34587
--builtin_ntp_servers=127.0.194.212:37287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 03:48:44.922555 18546 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:48:44.922958 18546 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:48:44.923121 18546 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:48:44.930462 18546 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:48:44.930614 18546 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:48:44.930872 18546 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.194.193
I20260926 03:48:44.940449 18546 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.194.212:37287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790394221564982-779-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790394221564982-779-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.194.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790394221564982-779-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.194.193
--webserver_port=0
--tserver_master_addrs=127.0.194.254:34587
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.194.193
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790394221564982-779-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:48:44.941782 18546 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:48:44.943588 18546 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:48:44.956807 18551 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:48:44.956861 18554 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:48:44.956831 18552 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:48:44.957719 18546 server_base.cc:1034] running on GCE node
I20260926 03:48:44.958370 18546 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:48:44.959422 18546 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:48:44.960706 18546 hybrid_clock.cc:648] HybridClock initialized: now 1790394524960648 us; error 47 us; skew 500 ppm
I20260926 03:48:44.961103 18546 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 03:48:44.963614 18546 webserver.cc:460] Webserver started at http://127.0.194.193:34667/ using document root <none> and password file <none>
I20260926 03:48:44.964372 18546 fs_manager.cc:359] Metadata directory not provided
I20260926 03:48:44.964509 18546 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:48:44.964852 18546 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:48:44.967902 18546 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790394221564982-779-0/minicluster-data/ts-0/data/instance:
uuid: "427a1f55f39c420f88edc96de2f3948b"
format_stamp: "Formatted at 2026-09-26 03:48:44 on dist-test-slave-n2th"
server_key: "6843f52c0775d3a612fde87c1b4c8dec"
server_key_iv: "f30e29d44332187e5979dce5dbf283cc"
server_key_version: "encryptionkey@0"
I20260926 03:48:44.968720 18546 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790394221564982-779-0/minicluster-data/ts-0/wal/instance:
uuid: "427a1f55f39c420f88edc96de2f3948b"
format_stamp: "Formatted at 2026-09-26 03:48:44 on dist-test-slave-n2th"
server_key: "6843f52c0775d3a612fde87c1b4c8dec"
server_key_iv: "f30e29d44332187e5979dce5dbf283cc"
server_key_version: "encryptionkey@0"
I20260926 03:48:44.974119 18546 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 03:48:44.977857 18560 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:48:44.979318 18546 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 03:48:44.979506 18546 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790394221564982-779-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790394221564982-779-0/minicluster-data/ts-0/wal
uuid: "427a1f55f39c420f88edc96de2f3948b"
format_stamp: "Formatted at 2026-09-26 03:48:44 on dist-test-slave-n2th"
server_key: "6843f52c0775d3a612fde87c1b4c8dec"
server_key_iv: "f30e29d44332187e5979dce5dbf283cc"
server_key_version: "encryptionkey@0"
I20260926 03:48:44.979753 18546 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790394221564982-779-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790394221564982-779-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790394221564982-779-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:48:44.995344 18546 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:48:44.996232 18546 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:48:44.996546 18546 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:48:44.997532 18546 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:48:44.999151 18546 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:48:44.999272 18546 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:48:44.999402 18546 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:48:44.999488 18546 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:48:45.034965 18546 rpc_server.cc:307] RPC server started. Bound to: 127.0.194.193:45861
I20260926 03:48:45.035023 18672 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.194.193:45861 every 8 connection(s)
I20260926 03:48:45.036319 18546 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790394221564982-779-0/minicluster-data/ts-0/data/info.pb
I20260926 03:48:45.044834 779 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu as pid 18546
I20260926 03:48:45.045010 779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790394221564982-779-0/minicluster-data/ts-0/wal/instance
I20260926 03:48:45.045468 779 external_mini_cluster.cc:1428] Setting key 4269df062d5ff98c38d7c2563166a7c6
I20260926 03:48:45.048629 779 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
/tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790394221564982-779-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790394221564982-779-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790394221564982-779-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790394221564982-779-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.194.194:0
--local_ip_for_outbound_sockets=127.0.194.194
--webserver_interface=127.0.194.194
--webserver_port=0
--tserver_master_addrs=127.0.194.254:34587
--builtin_ntp_servers=127.0.194.212:37287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 03:48:45.049425 18675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:45.038976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.193:56117 (local address 127.0.194.254:34587)
0926 03:48:45.039407 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0926 03:48:45.039412 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:48:45.040617 (+ 1205us) server_negotiation.cc:407] Connection header received
0926 03:48:45.042437 (+ 1820us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:48:45.042443 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:48:45.042537 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:48:45.042659 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:48:45.043772 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:48:45.044569 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:48:45.046343 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:48:45.046889 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:48:45.046997 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:48:45.048483 (+ 1486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:48:45.048490 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:48:45.048494 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:48:45.048517 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:48:45.048568 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:48:45.048587 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:48:45.048689 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 03:48:45.048992 (+ 303us) server_negotiation.cc:299] Negotiation successful
0926 03:48:45.049090 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":230,"threads_started":1}
I20260926 03:48:45.050499 18676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:45.039591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:34587 (local address 127.0.194.193:56117)
0926 03:48:45.040461 (+ 870us) negotiation.cc:107] Waiting for socket to connect
0926 03:48:45.040502 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 03:48:45.042154 (+ 1652us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:48:45.042925 (+ 771us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:48:45.042934 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:48:45.043026 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:48:45.043600 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:48:45.043617 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:48:45.044759 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:48:45.044763 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:48:45.046196 (+ 1433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:48:45.046225 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:48:45.047125 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:48:45.047133 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:48:45.047398 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:48:45.048135 (+ 737us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 03:48:45.048138 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:48:45.048178 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 03:48:45.048257 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:48:45.048693 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:48:45.048702 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:48:45.048708 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 03:48:45.048937 (+ 229us) client_negotiation.cc:240] Negotiation successful
0926 03:48:45.049371 (+ 434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":262,"threads_started":1}
I20260926 03:48:45.052285 18673 heartbeater.cc:347] Connected to a master server at 127.0.194.254:34587
I20260926 03:48:45.052663 18673 heartbeater.cc:464] Registering TS with master...
I20260926 03:48:45.053490 18673 heartbeater.cc:511] Master 127.0.194.254:34587 requested a full tablet report, sending...
I20260926 03:48:45.055918 18487 ts_manager.cc:194] Registered new tserver with Master: 427a1f55f39c420f88edc96de2f3948b (127.0.194.193:45861)
I20260926 03:48:45.057744 18487 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.194.193:56117
W20260926 03:48:45.385952 18677 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:48:45.386236 18677 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:48:45.386384 18677 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:48:45.393831 18677 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:48:45.393944 18677 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:48:45.394131 18677 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.194.194
I20260926 03:48:45.403641 18677 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.194.212:37287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790394221564982-779-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790394221564982-779-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.194.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790394221564982-779-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.194.194
--webserver_port=0
--tserver_master_addrs=127.0.194.254:34587
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.194.194
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790394221564982-779-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:48:45.404986 18677 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:48:45.406950 18677 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:48:45.420310 18682 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:48:45.420346 18683 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:48:45.420692 18685 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:48:45.421327 18677 server_base.cc:1034] running on GCE node
I20260926 03:48:45.422005 18677 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:48:45.422976 18677 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:48:45.424185 18677 hybrid_clock.cc:648] HybridClock initialized: now 1790394525424131 us; error 47 us; skew 500 ppm
I20260926 03:48:45.424480 18677 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 03:48:45.426873 18677 webserver.cc:460] Webserver started at http://127.0.194.194:43449/ using document root <none> and password file <none>
I20260926 03:48:45.427656 18677 fs_manager.cc:359] Metadata directory not provided
I20260926 03:48:45.427790 18677 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:48:45.428115 18677 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:48:45.431021 18677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790394221564982-779-0/minicluster-data/ts-1/data/instance:
uuid: "e02aa6b086c1471d975a17a8ab244b82"
format_stamp: "Formatted at 2026-09-26 03:48:45 on dist-test-slave-n2th"
server_key: "5fdda6aa9a630f175042a4bacb04b9a6"
server_key_iv: "1d647f9d3ea8a55960297a96cc7497e6"
server_key_version: "encryptionkey@0"
I20260926 03:48:45.431806 18677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790394221564982-779-0/minicluster-data/ts-1/wal/instance:
uuid: "e02aa6b086c1471d975a17a8ab244b82"
format_stamp: "Formatted at 2026-09-26 03:48:45 on dist-test-slave-n2th"
server_key: "5fdda6aa9a630f175042a4bacb04b9a6"
server_key_iv: "1d647f9d3ea8a55960297a96cc7497e6"
server_key_version: "encryptionkey@0"
I20260926 03:48:45.436941 18677 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260926 03:48:45.440490 18691 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:48:45.441771 18677 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 03:48:45.441941 18677 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790394221564982-779-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790394221564982-779-0/minicluster-data/ts-1/wal
uuid: "e02aa6b086c1471d975a17a8ab244b82"
format_stamp: "Formatted at 2026-09-26 03:48:45 on dist-test-slave-n2th"
server_key: "5fdda6aa9a630f175042a4bacb04b9a6"
server_key_iv: "1d647f9d3ea8a55960297a96cc7497e6"
server_key_version: "encryptionkey@0"
I20260926 03:48:45.442155 18677 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790394221564982-779-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790394221564982-779-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790394221564982-779-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:48:45.462178 18677 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:48:45.463301 18677 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:48:45.463618 18677 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:48:45.464623 18677 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:48:45.466143 18677 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:48:45.466266 18677 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:48:45.466388 18677 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:48:45.466492 18677 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:48:45.499991 18677 rpc_server.cc:307] RPC server started. Bound to: 127.0.194.194:45735
I20260926 03:48:45.500108 18803 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.194.194:45735 every 8 connection(s)
I20260926 03:48:45.501330 18677 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790394221564982-779-0/minicluster-data/ts-1/data/info.pb
I20260926 03:48:45.507572 779 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu as pid 18677
I20260926 03:48:45.507786 779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790394221564982-779-0/minicluster-data/ts-1/wal/instance
I20260926 03:48:45.508279 779 external_mini_cluster.cc:1428] Setting key 75f78c80b049253d7a688e90e12e938c
I20260926 03:48:45.511231 779 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
/tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790394221564982-779-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790394221564982-779-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790394221564982-779-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790394221564982-779-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.194.195:0
--local_ip_for_outbound_sockets=127.0.194.195
--webserver_interface=127.0.194.195
--webserver_port=0
--tserver_master_addrs=127.0.194.254:34587
--builtin_ntp_servers=127.0.194.212:37287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 03:48:45.514755 18675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:45.503877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.194:40607 (local address 127.0.194.254:34587)
0926 03:48:45.504066 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0926 03:48:45.504070 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:48:45.505444 (+ 1374us) server_negotiation.cc:407] Connection header received
0926 03:48:45.507388 (+ 1944us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:48:45.507396 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:48:45.507498 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:48:45.507624 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:48:45.508917 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:48:45.509643 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:48:45.511616 (+ 1973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:48:45.512319 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:48:45.512486 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:48:45.513806 (+ 1320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:48:45.513816 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:48:45.513823 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:48:45.513851 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:48:45.513896 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:48:45.513913 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:48:45.514025 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0926 03:48:45.514231 (+ 206us) server_negotiation.cc:299] Negotiation successful
0926 03:48:45.514355 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 03:48:45.515687 18806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:45.504352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:34587 (local address 127.0.194.194:40607)
0926 03:48:45.505269 (+ 917us) negotiation.cc:107] Waiting for socket to connect
0926 03:48:45.505306 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 03:48:45.507080 (+ 1774us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:48:45.507934 (+ 854us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:48:45.507943 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:48:45.508038 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:48:45.508714 (+ 676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:48:45.508731 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:48:45.509800 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:48:45.509805 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:48:45.511430 (+ 1625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:48:45.511446 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:48:45.512542 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:48:45.512546 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:48:45.512747 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:48:45.513472 (+ 725us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 03:48:45.513475 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:48:45.513515 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 03:48:45.513599 (+ 84us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:48:45.514026 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:48:45.514033 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:48:45.514039 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 03:48:45.514181 (+ 142us) client_negotiation.cc:240] Negotiation successful
0926 03:48:45.514506 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":272,"threads_started":1}
I20260926 03:48:45.517194 18804 heartbeater.cc:347] Connected to a master server at 127.0.194.254:34587
I20260926 03:48:45.517535 18804 heartbeater.cc:464] Registering TS with master...
I20260926 03:48:45.518487 18804 heartbeater.cc:511] Master 127.0.194.254:34587 requested a full tablet report, sending...
I20260926 03:48:45.520279 18487 ts_manager.cc:194] Registered new tserver with Master: e02aa6b086c1471d975a17a8ab244b82 (127.0.194.194:45735)
I20260926 03:48:45.521145 18487 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.194.194:40607
W20260926 03:48:45.852682 18807 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:48:45.852980 18807 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:48:45.853133 18807 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:48:45.860582 18807 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:48:45.860708 18807 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:48:45.860890 18807 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.194.195
I20260926 03:48:45.870476 18807 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.194.212:37287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790394221564982-779-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790394221564982-779-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.194.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790394221564982-779-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.194.195
--webserver_port=0
--tserver_master_addrs=127.0.194.254:34587
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.194.195
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790394221564982-779-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:48:45.871847 18807 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:48:45.873524 18807 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:48:45.886762 18813 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:48:45.886680 18812 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:48:45.886638 18815 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:48:45.887583 18807 server_base.cc:1034] running on GCE node
I20260926 03:48:45.888288 18807 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:48:45.889238 18807 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:48:45.890503 18807 hybrid_clock.cc:648] HybridClock initialized: now 1790394525890456 us; error 62 us; skew 500 ppm
I20260926 03:48:45.890894 18807 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 03:48:45.893281 18807 webserver.cc:460] Webserver started at http://127.0.194.195:40931/ using document root <none> and password file <none>
I20260926 03:48:45.893977 18807 fs_manager.cc:359] Metadata directory not provided
I20260926 03:48:45.894101 18807 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:48:45.894444 18807 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:48:45.897334 18807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790394221564982-779-0/minicluster-data/ts-2/data/instance:
uuid: "f645405a37744d2db84d07099b6544fe"
format_stamp: "Formatted at 2026-09-26 03:48:45 on dist-test-slave-n2th"
server_key: "82aba406fd540d2cde7a36b7f6749232"
server_key_iv: "58f16368fbc87b8da1345e8784f32666"
server_key_version: "encryptionkey@0"
I20260926 03:48:45.898111 18807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790394221564982-779-0/minicluster-data/ts-2/wal/instance:
uuid: "f645405a37744d2db84d07099b6544fe"
format_stamp: "Formatted at 2026-09-26 03:48:45 on dist-test-slave-n2th"
server_key: "82aba406fd540d2cde7a36b7f6749232"
server_key_iv: "58f16368fbc87b8da1345e8784f32666"
server_key_version: "encryptionkey@0"
I20260926 03:48:45.903956 18807 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260926 03:48:45.907567 18821 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:48:45.909009 18807 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 03:48:45.909193 18807 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790394221564982-779-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790394221564982-779-0/minicluster-data/ts-2/wal
uuid: "f645405a37744d2db84d07099b6544fe"
format_stamp: "Formatted at 2026-09-26 03:48:45 on dist-test-slave-n2th"
server_key: "82aba406fd540d2cde7a36b7f6749232"
server_key_iv: "58f16368fbc87b8da1345e8784f32666"
server_key_version: "encryptionkey@0"
I20260926 03:48:45.909425 18807 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790394221564982-779-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790394221564982-779-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790394221564982-779-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:48:45.928444 18807 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:48:45.929353 18807 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:48:45.929683 18807 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:48:45.930692 18807 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:48:45.932327 18807 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:48:45.932417 18807 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:48:45.932492 18807 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:48:45.932545 18807 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:48:45.965651 18807 rpc_server.cc:307] RPC server started. Bound to: 127.0.194.195:39947
I20260926 03:48:45.965728 18933 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.194.195:39947 every 8 connection(s)
I20260926 03:48:45.967085 18807 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790394221564982-779-0/minicluster-data/ts-2/data/info.pb
I20260926 03:48:45.972914 779 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu as pid 18807
I20260926 03:48:45.973119 779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790394221564982-779-0/minicluster-data/ts-2/wal/instance
I20260926 03:48:45.973662 779 external_mini_cluster.cc:1428] Setting key a8818e2cd77e2706f4501c9ddc5eb818
I20260926 03:48:45.979833 18675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:45.969632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.195:41815 (local address 127.0.194.254:34587)
0926 03:48:45.969866 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0926 03:48:45.969873 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:48:45.971311 (+ 1438us) server_negotiation.cc:407] Connection header received
0926 03:48:45.973092 (+ 1781us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:48:45.973099 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:48:45.973198 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:48:45.973319 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:48:45.974542 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:48:45.975665 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:48:45.977377 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:48:45.977788 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:48:45.977886 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:48:45.979198 (+ 1312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:48:45.979202 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:48:45.979205 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:48:45.979223 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:48:45.979251 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:48:45.979263 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:48:45.979332 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 03:48:45.979523 (+ 191us) server_negotiation.cc:299] Negotiation successful
0926 03:48:45.979601 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260926 03:48:45.980840 18936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:45.970207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:34587 (local address 127.0.194.195:41815)
0926 03:48:45.971156 (+ 949us) negotiation.cc:107] Waiting for socket to connect
0926 03:48:45.971193 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 03:48:45.972800 (+ 1607us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:48:45.973526 (+ 726us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:48:45.973535 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:48:45.973625 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:48:45.974385 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:48:45.974405 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:48:45.975846 (+ 1441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:48:45.975851 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:48:45.977245 (+ 1394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:48:45.977257 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:48:45.977971 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:48:45.977976 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:48:45.978167 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:48:45.978912 (+ 745us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 03:48:45.978915 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:48:45.978954 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 03:48:45.979039 (+ 85us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:48:45.979346 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:48:45.979353 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:48:45.979359 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 03:48:45.979499 (+ 140us) client_negotiation.cc:240] Negotiation successful
0926 03:48:45.979790 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":269,"threads_started":1}
I20260926 03:48:45.982154 18934 heartbeater.cc:347] Connected to a master server at 127.0.194.254:34587
I20260926 03:48:45.982467 18934 heartbeater.cc:464] Registering TS with master...
I20260926 03:48:45.983296 18934 heartbeater.cc:511] Master 127.0.194.254:34587 requested a full tablet report, sending...
I20260926 03:48:45.984792 18487 ts_manager.cc:194] Registered new tserver with Master: f645405a37744d2db84d07099b6544fe (127.0.194.195:39947)
I20260926 03:48:45.985626 18487 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.194.195:41815
I20260926 03:48:45.988986 779 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 03:48:45.998679 18675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:45.993046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53402 (local address 127.0.194.254:34587)
0926 03:48:45.993209 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0926 03:48:45.993213 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:48:45.993623 (+ 410us) server_negotiation.cc:407] Connection header received
0926 03:48:45.993756 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:48:45.993760 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:48:45.993832 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:48:45.993917 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:48:45.994430 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:48:45.995186 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:48:45.996510 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:48:45.996908 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:48:45.996998 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:48:45.998108 (+ 1110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:48:45.998113 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:48:45.998115 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:48:45.998131 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:48:45.998157 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:48:45.998167 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:48:45.998232 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 03:48:45.998373 (+ 141us) server_negotiation.cc:299] Negotiation successful
0926 03:48:45.998445 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 03:48:46.002707 18487 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53402:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 03:48:46.009898 18487 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 03:48:46.041260 18949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:46.023797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43326 (local address 127.0.194.194:45735)
0926 03:48:46.030796 (+ 6999us) server_negotiation.cc:206] Beginning negotiation
0926 03:48:46.030805 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:48:46.031782 (+ 977us) server_negotiation.cc:407] Connection header received
0926 03:48:46.032168 (+ 386us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:48:46.032176 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:48:46.032500 (+ 324us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:48:46.032690 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:48:46.033821 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:48:46.035413 (+ 1592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:48:46.038204 (+ 2791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:48:46.039701 (+ 1497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:48:46.039808 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:48:46.040068 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0926 03:48:46.040708 (+ 640us) server_negotiation.cc:299] Negotiation successful
0926 03:48:46.040904 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6804,"thread_start_us":7122,"threads_started":1}
I20260926 03:48:46.042248 18946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:46.023256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.194:45735 (local address 127.0.0.1:43326)
0926 03:48:46.031665 (+ 8409us) negotiation.cc:107] Waiting for socket to connect
0926 03:48:46.031688 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 03:48:46.031909 (+ 221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:48:46.033041 (+ 1132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:48:46.033049 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:48:46.033083 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:48:46.033572 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:48:46.033587 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:48:46.035616 (+ 2029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:48:46.035621 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:48:46.037962 (+ 2341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:48:46.037978 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:48:46.040253 (+ 2275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:48:46.040259 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:48:46.040450 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:48:46.040496 (+ 46us) client_negotiation.cc:769] Sending connection context
0926 03:48:46.041629 (+ 1133us) client_negotiation.cc:240] Negotiation successful
0926 03:48:46.041819 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8253,"spinlock_wait_cycles":13952,"thread_start_us":775,"threads_started":1}
I20260926 03:48:46.044315 18948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:46.022409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.195:39947 (local address 127.0.0.1:57900)
0926 03:48:46.023971 (+ 1562us) negotiation.cc:107] Waiting for socket to connect
0926 03:48:46.024016 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 03:48:46.024197 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:48:46.031822 (+ 7625us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:48:46.031829 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:48:46.031884 (+ 55us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:48:46.032258 (+ 374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:48:46.032274 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:48:46.036319 (+ 4045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:48:46.036327 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:48:46.039280 (+ 2953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:48:46.039300 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:48:46.043321 (+ 4021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:48:46.043330 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:48:46.043645 (+ 315us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:48:46.043699 (+ 54us) client_negotiation.cc:769] Sending connection context
0926 03:48:46.043777 (+ 78us) client_negotiation.cc:240] Negotiation successful
0926 03:48:46.043936 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1289,"spinlock_wait_cycles":1792,"thread_start_us":8966,"threads_started":1}
I20260926 03:48:46.045523 18947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:46.022756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57900 (local address 127.0.194.195:39947)
0926 03:48:46.031282 (+ 8526us) server_negotiation.cc:206] Beginning negotiation
0926 03:48:46.031288 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:48:46.031317 (+ 29us) server_negotiation.cc:407] Connection header received
0926 03:48:46.031398 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:48:46.031404 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:48:46.031601 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:48:46.031812 (+ 211us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:48:46.032502 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:48:46.036132 (+ 3630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:48:46.041136 (+ 5004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:48:46.043044 (+ 1908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:48:46.044626 (+ 1582us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:48:46.044921 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0926 03:48:46.044998 (+ 77us) server_negotiation.cc:299] Negotiation successful
0926 03:48:46.045167 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8375,"thread_start_us":8214,"threads_started":1}
I20260926 03:48:46.047154 18739 tablet_service.cc:1467] Processing CreateTablet for tablet 510aada2501a45eeb5f2430547f8f5b9 (DEFAULT_TABLE table=test-table [id=343cb8e6f91645a9a1f659dbd3b11d66]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:48:46.048549 18739 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 510aada2501a45eeb5f2430547f8f5b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:48:46.049844 18869 tablet_service.cc:1467] Processing CreateTablet for tablet 510aada2501a45eeb5f2430547f8f5b9 (DEFAULT_TABLE table=test-table [id=343cb8e6f91645a9a1f659dbd3b11d66]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:48:46.051146 18869 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 510aada2501a45eeb5f2430547f8f5b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:48:46.056392 18946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:46.031838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.193:45861 (local address 127.0.0.1:57880)
0926 03:48:46.044157 (+ 12319us) negotiation.cc:107] Waiting for socket to connect
0926 03:48:46.044185 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0926 03:48:46.044325 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:48:46.049590 (+ 5265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:48:46.049596 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:48:46.049624 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:48:46.049794 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:48:46.049805 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:48:46.051738 (+ 1933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:48:46.051744 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:48:46.053597 (+ 1853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:48:46.053613 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:48:46.055578 (+ 1965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:48:46.055584 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:48:46.055790 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:48:46.055831 (+ 41us) client_negotiation.cc:769] Sending connection context
0926 03:48:46.055937 (+ 106us) client_negotiation.cc:240] Negotiation successful
0926 03:48:46.056093 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12176,"thread_start_us":12781,"threads_started":1}
I20260926 03:48:46.056764 18951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:46.032441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57880 (local address 127.0.194.193:45861)
0926 03:48:46.048975 (+ 16534us) server_negotiation.cc:206] Beginning negotiation
0926 03:48:46.048984 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:48:46.049016 (+ 32us) server_negotiation.cc:407] Connection header received
0926 03:48:46.049112 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:48:46.049121 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:48:46.049361 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:48:46.049570 (+ 209us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:48:46.049977 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:48:46.051580 (+ 1603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:48:46.053828 (+ 2248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:48:46.055397 (+ 1569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:48:46.055546 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:48:46.055814 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0926 03:48:46.056009 (+ 195us) server_negotiation.cc:299] Negotiation successful
0926 03:48:46.056181 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16328,"thread_start_us":17322,"threads_started":1}
I20260926 03:48:46.060705 18608 tablet_service.cc:1467] Processing CreateTablet for tablet 510aada2501a45eeb5f2430547f8f5b9 (DEFAULT_TABLE table=test-table [id=343cb8e6f91645a9a1f659dbd3b11d66]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:48:46.061659 18673 heartbeater.cc:503] Master 127.0.194.254:34587 was elected leader, sending a full tablet report...
I20260926 03:48:46.062953 18608 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 510aada2501a45eeb5f2430547f8f5b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:48:46.063051 18952 tablet_bootstrap.cc:492] T 510aada2501a45eeb5f2430547f8f5b9 P e02aa6b086c1471d975a17a8ab244b82: Bootstrap starting.
I20260926 03:48:46.065598 18953 tablet_bootstrap.cc:492] T 510aada2501a45eeb5f2430547f8f5b9 P f645405a37744d2db84d07099b6544fe: Bootstrap starting.
I20260926 03:48:46.066661 18952 tablet_bootstrap.cc:654] T 510aada2501a45eeb5f2430547f8f5b9 P e02aa6b086c1471d975a17a8ab244b82: Neither blocks nor log segments found. Creating new log.
I20260926 03:48:46.068048 18952 log.cc:824] T 510aada2501a45eeb5f2430547f8f5b9 P e02aa6b086c1471d975a17a8ab244b82: Log is configured to *not* fsync() on all Append() calls
I20260926 03:48:46.069290 18953 tablet_bootstrap.cc:654] T 510aada2501a45eeb5f2430547f8f5b9 P f645405a37744d2db84d07099b6544fe: Neither blocks nor log segments found. Creating new log.
I20260926 03:48:46.070513 18953 log.cc:824] T 510aada2501a45eeb5f2430547f8f5b9 P f645405a37744d2db84d07099b6544fe: Log is configured to *not* fsync() on all Append() calls
I20260926 03:48:46.071653 18952 tablet_bootstrap.cc:492] T 510aada2501a45eeb5f2430547f8f5b9 P e02aa6b086c1471d975a17a8ab244b82: No bootstrap required, opened a new log
I20260926 03:48:46.071998 18952 ts_tablet_manager.cc:1397] T 510aada2501a45eeb5f2430547f8f5b9 P e02aa6b086c1471d975a17a8ab244b82: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.001s
I20260926 03:48:46.074371 18953 tablet_bootstrap.cc:492] T 510aada2501a45eeb5f2430547f8f5b9 P f645405a37744d2db84d07099b6544fe: No bootstrap required, opened a new log
I20260926 03:48:46.074669 18953 ts_tablet_manager.cc:1397] T 510aada2501a45eeb5f2430547f8f5b9 P f645405a37744d2db84d07099b6544fe: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.004s
I20260926 03:48:46.076011 18956 tablet_bootstrap.cc:492] T 510aada2501a45eeb5f2430547f8f5b9 P 427a1f55f39c420f88edc96de2f3948b: Bootstrap starting.
I20260926 03:48:46.078984 18956 tablet_bootstrap.cc:654] T 510aada2501a45eeb5f2430547f8f5b9 P 427a1f55f39c420f88edc96de2f3948b: Neither blocks nor log segments found. Creating new log.
I20260926 03:48:46.079955 18956 log.cc:824] T 510aada2501a45eeb5f2430547f8f5b9 P 427a1f55f39c420f88edc96de2f3948b: Log is configured to *not* fsync() on all Append() calls
I20260926 03:48:46.083125 18952 raft_consensus.cc:348] T 510aada2501a45eeb5f2430547f8f5b9 P e02aa6b086c1471d975a17a8ab244b82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f645405a37744d2db84d07099b6544fe" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 39947 } } peers { permanent_uuid: "427a1f55f39c420f88edc96de2f3948b" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 45861 } } peers { permanent_uuid: "e02aa6b086c1471d975a17a8ab244b82" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 45735 } }
I20260926 03:48:46.083583 18956 tablet_bootstrap.cc:492] T 510aada2501a45eeb5f2430547f8f5b9 P 427a1f55f39c420f88edc96de2f3948b: No bootstrap required, opened a new log
I20260926 03:48:46.083590 18952 raft_consensus.cc:374] T 510aada2501a45eeb5f2430547f8f5b9 P e02aa6b086c1471d975a17a8ab244b82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:48:46.083736 18952 raft_consensus.cc:729] T 510aada2501a45eeb5f2430547f8f5b9 P e02aa6b086c1471d975a17a8ab244b82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e02aa6b086c1471d975a17a8ab244b82, State: Initialized, Role: FOLLOWER
I20260926 03:48:46.083830 18956 ts_tablet_manager.cc:1397] T 510aada2501a45eeb5f2430547f8f5b9 P 427a1f55f39c420f88edc96de2f3948b: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.006s
I20260926 03:48:46.084393 18952 consensus_queue.cc:260] T 510aada2501a45eeb5f2430547f8f5b9 P e02aa6b086c1471d975a17a8ab244b82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f645405a37744d2db84d07099b6544fe" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 39947 } } peers { permanent_uuid: "427a1f55f39c420f88edc96de2f3948b" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 45861 } } peers { permanent_uuid: "e02aa6b086c1471d975a17a8ab244b82" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 45735 } }
I20260926 03:48:46.085446 18953 raft_consensus.cc:348] T 510aada2501a45eeb5f2430547f8f5b9 P f645405a37744d2db84d07099b6544fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f645405a37744d2db84d07099b6544fe" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 39947 } } peers { permanent_uuid: "427a1f55f39c420f88edc96de2f3948b" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 45861 } } peers { permanent_uuid: "e02aa6b086c1471d975a17a8ab244b82" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 45735 } }
I20260926 03:48:46.086014 18953 raft_consensus.cc:374] T 510aada2501a45eeb5f2430547f8f5b9 P f645405a37744d2db84d07099b6544fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:48:46.086138 18953 raft_consensus.cc:729] T 510aada2501a45eeb5f2430547f8f5b9 P f645405a37744d2db84d07099b6544fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f645405a37744d2db84d07099b6544fe, State: Initialized, Role: FOLLOWER
I20260926 03:48:46.086941 18953 consensus_queue.cc:260] T 510aada2501a45eeb5f2430547f8f5b9 P f645405a37744d2db84d07099b6544fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f645405a37744d2db84d07099b6544fe" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 39947 } } peers { permanent_uuid: "427a1f55f39c420f88edc96de2f3948b" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 45861 } } peers { permanent_uuid: "e02aa6b086c1471d975a17a8ab244b82" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 45735 } }
I20260926 03:48:46.091780 18934 heartbeater.cc:503] Master 127.0.194.254:34587 was elected leader, sending a full tablet report...
I20260926 03:48:46.092352 18956 raft_consensus.cc:348] T 510aada2501a45eeb5f2430547f8f5b9 P 427a1f55f39c420f88edc96de2f3948b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f645405a37744d2db84d07099b6544fe" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 39947 } } peers { permanent_uuid: "427a1f55f39c420f88edc96de2f3948b" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 45861 } } peers { permanent_uuid: "e02aa6b086c1471d975a17a8ab244b82" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 45735 } }
I20260926 03:48:46.092736 18953 ts_tablet_manager.cc:1428] T 510aada2501a45eeb5f2430547f8f5b9 P f645405a37744d2db84d07099b6544fe: Time spent starting tablet: real 0.018s user 0.006s sys 0.010s
I20260926 03:48:46.092763 18956 raft_consensus.cc:374] T 510aada2501a45eeb5f2430547f8f5b9 P 427a1f55f39c420f88edc96de2f3948b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:48:46.092850 18956 raft_consensus.cc:729] T 510aada2501a45eeb5f2430547f8f5b9 P 427a1f55f39c420f88edc96de2f3948b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 427a1f55f39c420f88edc96de2f3948b, State: Initialized, Role: FOLLOWER
I20260926 03:48:46.093544 18956 consensus_queue.cc:260] T 510aada2501a45eeb5f2430547f8f5b9 P 427a1f55f39c420f88edc96de2f3948b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f645405a37744d2db84d07099b6544fe" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 39947 } } peers { permanent_uuid: "427a1f55f39c420f88edc96de2f3948b" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 45861 } } peers { permanent_uuid: "e02aa6b086c1471d975a17a8ab244b82" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 45735 } }
I20260926 03:48:46.095273 18956 ts_tablet_manager.cc:1428] T 510aada2501a45eeb5f2430547f8f5b9 P 427a1f55f39c420f88edc96de2f3948b: Time spent starting tablet: real 0.011s user 0.004s sys 0.006s
I20260926 03:48:46.096484 18804 heartbeater.cc:503] Master 127.0.194.254:34587 was elected leader, sending a full tablet report...
I20260926 03:48:46.096797 18952 ts_tablet_manager.cc:1428] T 510aada2501a45eeb5f2430547f8f5b9 P e02aa6b086c1471d975a17a8ab244b82: Time spent starting tablet: real 0.025s user 0.011s sys 0.013s
I20260926 03:48:46.215916 18960 raft_consensus.cc:482] T 510aada2501a45eeb5f2430547f8f5b9 P 427a1f55f39c420f88edc96de2f3948b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 03:48:46.216253 18960 raft_consensus.cc:504] T 510aada2501a45eeb5f2430547f8f5b9 P 427a1f55f39c420f88edc96de2f3948b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f645405a37744d2db84d07099b6544fe" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 39947 } } peers { permanent_uuid: "427a1f55f39c420f88edc96de2f3948b" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 45861 } } peers { permanent_uuid: "e02aa6b086c1471d975a17a8ab244b82" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 45735 } }
I20260926 03:48:46.217938 18960 leader_election.cc:288] T 510aada2501a45eeb5f2430547f8f5b9 P 427a1f55f39c420f88edc96de2f3948b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f645405a37744d2db84d07099b6544fe (127.0.194.195:39947), e02aa6b086c1471d975a17a8ab244b82 (127.0.194.194:45735)
W20260926 03:48:46.218894 18935 tablet.cc:2369] T 510aada2501a45eeb5f2430547f8f5b9 P f645405a37744d2db84d07099b6544fe: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 03:48:46.225106 18949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:46.218494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.193:43747 (local address 127.0.194.194:45735)
0926 03:48:46.219079 (+ 585us) server_negotiation.cc:206] Beginning negotiation
0926 03:48:46.219085 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:48:46.219630 (+ 545us) server_negotiation.cc:407] Connection header received
0926 03:48:46.219783 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:48:46.219792 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:48:46.219892 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:48:46.220013 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:48:46.220686 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:48:46.221350 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:48:46.223393 (+ 2043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:48:46.224138 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:48:46.224311 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:48:46.224380 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 03:48:46.224710 (+ 330us) server_negotiation.cc:299] Negotiation successful
0926 03:48:46.224832 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":426}
I20260926 03:48:46.225113 18947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:46.218210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.193:36313 (local address 127.0.194.195:39947)
0926 03:48:46.218411 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0926 03:48:46.218416 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:48:46.219099 (+ 683us) server_negotiation.cc:407] Connection header received
0926 03:48:46.219485 (+ 386us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:48:46.219491 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:48:46.219579 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:48:46.219644 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:48:46.220630 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:48:46.221310 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:48:46.223409 (+ 2099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:48:46.224167 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:48:46.224350 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:48:46.224415 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 03:48:46.224764 (+ 349us) server_negotiation.cc:299] Negotiation successful
0926 03:48:46.224886 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 03:48:46.225257 18961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:46.218211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.195:39947 (local address 127.0.194.193:36313)
0926 03:48:46.218996 (+ 785us) negotiation.cc:107] Waiting for socket to connect
0926 03:48:46.219017 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 03:48:46.219237 (+ 220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:48:46.219858 (+ 621us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:48:46.219861 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:48:46.219880 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:48:46.220448 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:48:46.220461 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:48:46.221440 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:48:46.221449 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:48:46.223214 (+ 1765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:48:46.223226 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:48:46.224363 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:48:46.224370 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:48:46.224601 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:48:46.224624 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 03:48:46.224724 (+ 100us) client_negotiation.cc:240] Negotiation successful
0926 03:48:46.224855 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":642,"mutex_wait_us":105,"thread_start_us":416,"threads_started":1}
I20260926 03:48:46.225309 18962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:46.218364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.194:45735 (local address 127.0.194.193:43747)
0926 03:48:46.219522 (+ 1158us) negotiation.cc:107] Waiting for socket to connect
0926 03:48:46.219538 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 03:48:46.219649 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:48:46.220052 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:48:46.220088 (+ 36us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:48:46.220115 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:48:46.220515 (+ 400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:48:46.220524 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:48:46.221480 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:48:46.221487 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:48:46.223191 (+ 1704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:48:46.223206 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:48:46.224319 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:48:46.224325 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:48:46.224526 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:48:46.224582 (+ 56us) client_negotiation.cc:769] Sending connection context
0926 03:48:46.224683 (+ 101us) client_negotiation.cc:240] Negotiation successful
0926 03:48:46.224811 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1040,"mutex_wait_us":145,"thread_start_us":876,"threads_started":1}
I20260926 03:48:46.226280 18889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "510aada2501a45eeb5f2430547f8f5b9" candidate_uuid: "427a1f55f39c420f88edc96de2f3948b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f645405a37744d2db84d07099b6544fe" is_pre_election: true
I20260926 03:48:46.226281 18759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "510aada2501a45eeb5f2430547f8f5b9" candidate_uuid: "427a1f55f39c420f88edc96de2f3948b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e02aa6b086c1471d975a17a8ab244b82" is_pre_election: true
I20260926 03:48:46.226910 18759 raft_consensus.cc:2454] T 510aada2501a45eeb5f2430547f8f5b9 P e02aa6b086c1471d975a17a8ab244b82 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 427a1f55f39c420f88edc96de2f3948b in term 0.
I20260926 03:48:46.226910 18889 raft_consensus.cc:2454] T 510aada2501a45eeb5f2430547f8f5b9 P f645405a37744d2db84d07099b6544fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 427a1f55f39c420f88edc96de2f3948b in term 0.
I20260926 03:48:46.227698 18562 leader_election.cc:302] T 510aada2501a45eeb5f2430547f8f5b9 P 427a1f55f39c420f88edc96de2f3948b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 427a1f55f39c420f88edc96de2f3948b, f645405a37744d2db84d07099b6544fe; no voters:
I20260926 03:48:46.228148 18960 raft_consensus.cc:2781] T 510aada2501a45eeb5f2430547f8f5b9 P 427a1f55f39c420f88edc96de2f3948b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 03:48:46.228302 18960 raft_consensus.cc:482] T 510aada2501a45eeb5f2430547f8f5b9 P 427a1f55f39c420f88edc96de2f3948b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 03:48:46.228428 18960 raft_consensus.cc:3037] T 510aada2501a45eeb5f2430547f8f5b9 P 427a1f55f39c420f88edc96de2f3948b [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:48:46.230588 18960 raft_consensus.cc:504] T 510aada2501a45eeb5f2430547f8f5b9 P 427a1f55f39c420f88edc96de2f3948b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f645405a37744d2db84d07099b6544fe" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 39947 } } peers { permanent_uuid: "427a1f55f39c420f88edc96de2f3948b" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 45861 } } peers { permanent_uuid: "e02aa6b086c1471d975a17a8ab244b82" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 45735 } }
I20260926 03:48:46.231194 18960 leader_election.cc:288] T 510aada2501a45eeb5f2430547f8f5b9 P 427a1f55f39c420f88edc96de2f3948b [CANDIDATE]: Term 1 election: Requested vote from peers f645405a37744d2db84d07099b6544fe (127.0.194.195:39947), e02aa6b086c1471d975a17a8ab244b82 (127.0.194.194:45735)
I20260926 03:48:46.231652 18759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "510aada2501a45eeb5f2430547f8f5b9" candidate_uuid: "427a1f55f39c420f88edc96de2f3948b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e02aa6b086c1471d975a17a8ab244b82"
I20260926 03:48:46.231665 18889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "510aada2501a45eeb5f2430547f8f5b9" candidate_uuid: "427a1f55f39c420f88edc96de2f3948b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f645405a37744d2db84d07099b6544fe"
I20260926 03:48:46.231874 18889 raft_consensus.cc:3037] T 510aada2501a45eeb5f2430547f8f5b9 P f645405a37744d2db84d07099b6544fe [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:48:46.231858 18759 raft_consensus.cc:3037] T 510aada2501a45eeb5f2430547f8f5b9 P e02aa6b086c1471d975a17a8ab244b82 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:48:46.234042 18759 raft_consensus.cc:2454] T 510aada2501a45eeb5f2430547f8f5b9 P e02aa6b086c1471d975a17a8ab244b82 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 427a1f55f39c420f88edc96de2f3948b in term 1.
I20260926 03:48:46.234051 18889 raft_consensus.cc:2454] T 510aada2501a45eeb5f2430547f8f5b9 P f645405a37744d2db84d07099b6544fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 427a1f55f39c420f88edc96de2f3948b in term 1.
I20260926 03:48:46.234477 18564 leader_election.cc:302] T 510aada2501a45eeb5f2430547f8f5b9 P 427a1f55f39c420f88edc96de2f3948b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 427a1f55f39c420f88edc96de2f3948b, e02aa6b086c1471d975a17a8ab244b82; no voters:
I20260926 03:48:46.234808 18960 raft_consensus.cc:2781] T 510aada2501a45eeb5f2430547f8f5b9 P 427a1f55f39c420f88edc96de2f3948b [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:48:46.235268 18960 raft_consensus.cc:686] T 510aada2501a45eeb5f2430547f8f5b9 P 427a1f55f39c420f88edc96de2f3948b [term 1 LEADER]: Becoming Leader. State: Replica: 427a1f55f39c420f88edc96de2f3948b, State: Running, Role: LEADER
I20260926 03:48:46.235738 18960 consensus_queue.cc:237] T 510aada2501a45eeb5f2430547f8f5b9 P 427a1f55f39c420f88edc96de2f3948b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f645405a37744d2db84d07099b6544fe" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 39947 } } peers { permanent_uuid: "427a1f55f39c420f88edc96de2f3948b" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 45861 } } peers { permanent_uuid: "e02aa6b086c1471d975a17a8ab244b82" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 45735 } }
I20260926 03:48:46.240543 18485 catalog_manager.cc:5495] T 510aada2501a45eeb5f2430547f8f5b9 P 427a1f55f39c420f88edc96de2f3948b reported cstate change: term changed from 0 to 1, leader changed from <none> to 427a1f55f39c420f88edc96de2f3948b (127.0.194.193). New cstate: current_term: 1 leader_uuid: "427a1f55f39c420f88edc96de2f3948b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f645405a37744d2db84d07099b6544fe" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 39947 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "427a1f55f39c420f88edc96de2f3948b" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 45861 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e02aa6b086c1471d975a17a8ab244b82" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 45735 } health_report { overall_health: UNKNOWN } } }
W20260926 03:48:46.253424 18805 tablet.cc:2369] T 510aada2501a45eeb5f2430547f8f5b9 P e02aa6b086c1471d975a17a8ab244b82: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 03:48:46.273005 18951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:46.267819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57894 (local address 127.0.194.193:45861)
0926 03:48:46.268012 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0926 03:48:46.268017 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:48:46.268034 (+ 17us) server_negotiation.cc:407] Connection header received
0926 03:48:46.268222 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:48:46.268226 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:48:46.268328 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:48:46.268468 (+ 140us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:48:46.268751 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:48:46.269475 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:48:46.271088 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:48:46.271493 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:48:46.271586 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:48:46.272034 (+ 448us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:48:46.272552 (+ 518us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:48:46.272674 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0926 03:48:46.272753 (+ 79us) server_negotiation.cc:299] Negotiation successful
0926 03:48:46.272817 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 03:48:46.281049 18759 raft_consensus.cc:1261] T 510aada2501a45eeb5f2430547f8f5b9 P e02aa6b086c1471d975a17a8ab244b82 [term 1 FOLLOWER]: Refusing update from remote peer 427a1f55f39c420f88edc96de2f3948b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:48:46.281113 18889 raft_consensus.cc:1261] T 510aada2501a45eeb5f2430547f8f5b9 P f645405a37744d2db84d07099b6544fe [term 1 FOLLOWER]: Refusing update from remote peer 427a1f55f39c420f88edc96de2f3948b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:48:46.282065 18963 consensus_queue.cc:1035] T 510aada2501a45eeb5f2430547f8f5b9 P 427a1f55f39c420f88edc96de2f3948b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e02aa6b086c1471d975a17a8ab244b82" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 45735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:48:46.282434 18960 consensus_queue.cc:1035] T 510aada2501a45eeb5f2430547f8f5b9 P 427a1f55f39c420f88edc96de2f3948b [LEADER]: Connected to new peer: Peer: permanent_uuid: "f645405a37744d2db84d07099b6544fe" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 39947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 03:48:46.290094 18674 tablet.cc:2369] T 510aada2501a45eeb5f2430547f8f5b9 P 427a1f55f39c420f88edc96de2f3948b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 03:48:46.294365 18966 mvcc.cc:205] Tried to move back new op lower bound from 7333455979637379072 to 7333455979465596928. Current Snapshot: MvccSnapshot[applied={T|T < 7333455979637379072}]
I20260926 03:48:46.308713 18485 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:53402:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:48:46.309006 18485 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:53402:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:48:46.313043 18485 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6bdba152b50e4503a278636584aedcf0: Sending DeleteTablet for 3 replicas of tablet 510aada2501a45eeb5f2430547f8f5b9
I20260926 03:48:46.314390 18869 tablet_service.cc:1514] Processing DeleteTablet for tablet 510aada2501a45eeb5f2430547f8f5b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:48:46 UTC) from {username='slave'} at 127.0.0.1:57900
I20260926 03:48:46.314477 18608 tablet_service.cc:1514] Processing DeleteTablet for tablet 510aada2501a45eeb5f2430547f8f5b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:48:46 UTC) from {username='slave'} at 127.0.0.1:57880
I20260926 03:48:46.314502 18739 tablet_service.cc:1514] Processing DeleteTablet for tablet 510aada2501a45eeb5f2430547f8f5b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:48:46 UTC) from {username='slave'} at 127.0.0.1:43326
I20260926 03:48:46.316124 779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu pid 18546
I20260926 03:48:46.316476 18978 tablet_replica.cc:330] T 510aada2501a45eeb5f2430547f8f5b9 P e02aa6b086c1471d975a17a8ab244b82: stopping tablet replica
I20260926 03:48:46.316992 18978 raft_consensus.cc:2229] T 510aada2501a45eeb5f2430547f8f5b9 P e02aa6b086c1471d975a17a8ab244b82 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:48:46.317529 18978 raft_consensus.cc:2258] T 510aada2501a45eeb5f2430547f8f5b9 P e02aa6b086c1471d975a17a8ab244b82 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:48:46.320794 18978 ts_tablet_manager.cc:1905] T 510aada2501a45eeb5f2430547f8f5b9 P e02aa6b086c1471d975a17a8ab244b82: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:48:46.327279 18976 tablet_replica.cc:330] T 510aada2501a45eeb5f2430547f8f5b9 P 427a1f55f39c420f88edc96de2f3948b: stopping tablet replica
I20260926 03:48:46.327958 18976 raft_consensus.cc:2229] T 510aada2501a45eeb5f2430547f8f5b9 P 427a1f55f39c420f88edc96de2f3948b [term 1 LEADER]: Raft consensus shutting down.
I20260926 03:48:46.328837 18976 raft_consensus.cc:2258] T 510aada2501a45eeb5f2430547f8f5b9 P 427a1f55f39c420f88edc96de2f3948b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:48:46.329707 18978 ts_tablet_manager.cc:1918] T 510aada2501a45eeb5f2430547f8f5b9 P e02aa6b086c1471d975a17a8ab244b82: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:48:46.329741 18977 tablet_replica.cc:330] T 510aada2501a45eeb5f2430547f8f5b9 P f645405a37744d2db84d07099b6544fe: stopping tablet replica
I20260926 03:48:46.330188 18978 log.cc:1196] T 510aada2501a45eeb5f2430547f8f5b9 P e02aa6b086c1471d975a17a8ab244b82: Deleting WAL directory at /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790394221564982-779-0/minicluster-data/ts-1/wal/wals/510aada2501a45eeb5f2430547f8f5b9
I20260926 03:48:46.330243 18977 raft_consensus.cc:2229] T 510aada2501a45eeb5f2430547f8f5b9 P f645405a37744d2db84d07099b6544fe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:48:46.330857 18977 raft_consensus.cc:2258] T 510aada2501a45eeb5f2430547f8f5b9 P f645405a37744d2db84d07099b6544fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:48:46.331101 18978 ts_tablet_manager.cc:1939] T 510aada2501a45eeb5f2430547f8f5b9 P e02aa6b086c1471d975a17a8ab244b82: Deleting consensus metadata
I20260926 03:48:46.333279 18474 catalog_manager.cc:4841] TS e02aa6b086c1471d975a17a8ab244b82 (127.0.194.194:45735): tablet 510aada2501a45eeb5f2430547f8f5b9 (table test-table [id=343cb8e6f91645a9a1f659dbd3b11d66]) successfully deleted
I20260926 03:48:46.334055 18977 ts_tablet_manager.cc:1905] T 510aada2501a45eeb5f2430547f8f5b9 P f645405a37744d2db84d07099b6544fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:48:46.335943 18976 ts_tablet_manager.cc:1905] T 510aada2501a45eeb5f2430547f8f5b9 P 427a1f55f39c420f88edc96de2f3948b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:48:46.338460 18951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:46.317480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57908 (local address 127.0.194.193:45861)
0926 03:48:46.317665 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0926 03:48:46.317672 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:48:46.329635 (+ 11963us) server_negotiation.cc:407] Connection header received
0926 03:48:46.330012 (+ 377us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:48:46.330018 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:48:46.330145 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:48:46.330243 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:48:46.331055 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:48:46.332302 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:48:46.334145 (+ 1843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:48:46.334860 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:48:46.335010 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:48:46.336574 (+ 1564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:48:46.336592 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:48:46.336612 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:48:46.337477 (+ 865us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:48:46.337543 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:48:46.337574 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:48:46.337701 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0926 03:48:46.337934 (+ 233us) server_negotiation.cc:299] Negotiation successful
0926 03:48:46.338052 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 03:48:46.339691 18668 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:48:46.343789 18977 ts_tablet_manager.cc:1918] T 510aada2501a45eeb5f2430547f8f5b9 P f645405a37744d2db84d07099b6544fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:48:46.343984 18977 log.cc:1196] T 510aada2501a45eeb5f2430547f8f5b9 P f645405a37744d2db84d07099b6544fe: Deleting WAL directory at /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790394221564982-779-0/minicluster-data/ts-2/wal/wals/510aada2501a45eeb5f2430547f8f5b9
I20260926 03:48:46.344426 18977 ts_tablet_manager.cc:1939] T 510aada2501a45eeb5f2430547f8f5b9 P f645405a37744d2db84d07099b6544fe: Deleting consensus metadata
I20260926 03:48:46.346077 18472 catalog_manager.cc:4841] TS f645405a37744d2db84d07099b6544fe (127.0.194.195:39947): tablet 510aada2501a45eeb5f2430547f8f5b9 (table test-table [id=343cb8e6f91645a9a1f659dbd3b11d66]) successfully deleted
I20260926 03:48:46.438020 779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu with pid 18546
W20260926 03:48:46.456178 18472 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.194.193:45861 (error 108)
I20260926 03:48:46.456945 779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu pid 18677
I20260926 03:48:46.458364 18946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:46.457784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.193:45861 (local address 127.0.0.1:57910)
0926 03:48:46.458061 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0926 03:48:46.458173 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.193:45861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144}
W20260926 03:48:46.458632 18472 catalog_manager.cc:4584] TS 427a1f55f39c420f88edc96de2f3948b (127.0.194.193:45861): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 510aada2501a45eeb5f2430547f8f5b9: Network error: Client connection negotiation failed: client connection to 127.0.194.193:45861: connect: Connection refused (error 111)
I20260926 03:48:46.464272 18949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:46.457896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43328 (local address 127.0.194.194:45735)
0926 03:48:46.458057 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0926 03:48:46.458064 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:48:46.458084 (+ 20us) server_negotiation.cc:407] Connection header received
0926 03:48:46.458303 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:48:46.458308 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:48:46.458443 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:48:46.458538 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:48:46.459188 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:48:46.459938 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:48:46.461235 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:48:46.461640 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:48:46.461775 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:48:46.462949 (+ 1174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:48:46.462958 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:48:46.462972 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:48:46.463558 (+ 586us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:48:46.463605 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:48:46.463624 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:48:46.463753 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0926 03:48:46.463847 (+ 94us) server_negotiation.cc:299] Negotiation successful
0926 03:48:46.463957 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 03:48:46.465070 18799 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:48:46.512283 18946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:46.511698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.193:45861 (local address 127.0.0.1:57918)
0926 03:48:46.511960 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0926 03:48:46.512076 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.193:45861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260926 03:48:46.513561 18946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:46.513067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.193:45861 (local address 127.0.0.1:57920)
0926 03:48:46.513266 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0926 03:48:46.513378 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.193:45861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260926 03:48:46.560784 779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu with pid 18677
I20260926 03:48:46.574542 18946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:46.574054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.193:45861 (local address 127.0.0.1:57930)
0926 03:48:46.574300 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0926 03:48:46.574392 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.193:45861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260926 03:48:46.575841 18946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:46.575382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.193:45861 (local address 127.0.0.1:57934)
0926 03:48:46.575570 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0926 03:48:46.575669 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.193:45861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260926 03:48:46.584355 779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu pid 18807
I20260926 03:48:46.594568 18947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:46.585250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57902 (local address 127.0.194.195:39947)
0926 03:48:46.585447 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0926 03:48:46.585463 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 03:48:46.585574 (+ 111us) server_negotiation.cc:407] Connection header received
0926 03:48:46.585814 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:48:46.585823 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:48:46.585961 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:48:46.586103 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:48:46.586576 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:48:46.587896 (+ 1320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:48:46.590181 (+ 2285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:48:46.591012 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:48:46.591126 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:48:46.592554 (+ 1428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:48:46.592571 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:48:46.592594 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:48:46.593592 (+ 998us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:48:46.593671 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:48:46.593704 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:48:46.593834 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0926 03:48:46.593957 (+ 123us) server_negotiation.cc:299] Negotiation successful
0926 03:48:46.594095 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 03:48:46.595417 18929 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:48:46.656863 18946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:46.656229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.193:45861 (local address 127.0.0.1:57946)
0926 03:48:46.656520 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0926 03:48:46.656647 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.193:45861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260926 03:48:46.658223 18946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:46.657679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.193:45861 (local address 127.0.0.1:57952)
0926 03:48:46.657934 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0926 03:48:46.658027 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.193:45861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161}
I20260926 03:48:46.689337 779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu with pid 18807
I20260926 03:48:46.706636 779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu pid 18456
I20260926 03:48:46.707928 18517 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:48:46.830283 779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu with pid 18456
2026-09-26T03:48:46Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (2881 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790394221564982-779-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:48:46 dist-test-slave-n2th krb5kdc[18987](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:48:46 dist-test-slave-n2th krb5kdc[18987](info): set up 2 sockets
Sep 26 03:48:46 dist-test-slave-n2th krb5kdc[18987](info): commencing operation
krb5kdc: starting...
W20260926 03:48:48.011085 779 mini_kdc.cc:121] Time spent starting KDC: real 1.139s user 0.004s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790394221564982-779-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790394221564982-779-0/krb5kdc/test-user.keytab.
Sep 26 03:48:48 dist-test-slave-n2th krb5kdc[18987](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394528, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:48:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:48:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.194.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.194.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.194.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790394221564982-779-0/krb5kdc/kudu_127.0.194.254.keytab.
Entry for principal kudu/127.0.194.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790394221564982-779-0/krb5kdc/kudu_127.0.194.254.keytab.
I20260926 03:48:48.269665 779 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
/tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790394221564982-779-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790394221564982-779-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790394221564982-779-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790394221564982-779-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.194.254:45107
--webserver_interface=127.0.194.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790394221564982-779-0/krb5kdc/kudu_127.0.194.254.keytab
--principal=kudu/127.0.194.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.194.212:41949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.194.254:45107
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790394221564982-779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790394221564982-779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790394221564982-779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:48:48.611433 19000 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:48:48.611784 19000 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:48:48.611948 19000 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:48:48.619684 19000 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:48:48.619814 19000 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:48:48.619925 19000 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:48:48.620018 19000 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:48:48.620090 19000 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:48:48.630510 19000 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.194.212:41949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790394221564982-779-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790394221564982-779-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.194.254:45107
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790394221564982-779-0/krb5kdc/kudu_127.0.194.254.keytab
--principal=kudu/127.0.194.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.194.254:45107
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790394221564982-779-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.194.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790394221564982-779-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:48:48.632107 19000 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:48:48.633544 19000 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:48:48.645668 19005 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:48:48.645706 19008 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:48:48.645684 19006 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:48:48.648090 19000 server_base.cc:1034] running on GCE node
I20260926 03:48:48.648931 19000 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:48:48.650422 19000 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:48:48.651691 19000 hybrid_clock.cc:648] HybridClock initialized: now 1790394528651597 us; error 80 us; skew 500 ppm
I20260926 03:48:48.652112 19000 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:48:48 dist-test-slave-n2th krb5kdc[18987](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394528, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:48:48.655507 19000 init.cc:377] Logged in from keytab as kudu/127.0.194.254@KRBTEST.COM (short username kudu)
I20260926 03:48:48.657141 19000 webserver.cc:460] Webserver started at http://127.0.194.254:42803/ using document root <none> and password file <none>
I20260926 03:48:48.657862 19000 fs_manager.cc:359] Metadata directory not provided
I20260926 03:48:48.657963 19000 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:48:48.658234 19000 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:48:48.661346 19000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790394221564982-779-0/minicluster-data/master-0/data/instance:
uuid: "46d29631621547398f8f7e5451d9f243"
format_stamp: "Formatted at 2026-09-26 03:48:48 on dist-test-slave-n2th"
server_key: "935b542fa2d95ece72ad3cc5c33b76f2"
server_key_iv: "8bee6d6bc1a0836df91d376d66a455b8"
server_key_version: "encryptionkey@0"
I20260926 03:48:48.662096 19000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790394221564982-779-0/minicluster-data/master-0/wal/instance:
uuid: "46d29631621547398f8f7e5451d9f243"
format_stamp: "Formatted at 2026-09-26 03:48:48 on dist-test-slave-n2th"
server_key: "935b542fa2d95ece72ad3cc5c33b76f2"
server_key_iv: "8bee6d6bc1a0836df91d376d66a455b8"
server_key_version: "encryptionkey@0"
I20260926 03:48:48.667701 19000 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260926 03:48:48.671222 19015 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:48:48.672658 19000 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 03:48:48.672796 19000 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790394221564982-779-0/minicluster-data/master-0/data,/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790394221564982-779-0/minicluster-data/master-0/wal
uuid: "46d29631621547398f8f7e5451d9f243"
format_stamp: "Formatted at 2026-09-26 03:48:48 on dist-test-slave-n2th"
server_key: "935b542fa2d95ece72ad3cc5c33b76f2"
server_key_iv: "8bee6d6bc1a0836df91d376d66a455b8"
server_key_version: "encryptionkey@0"
I20260926 03:48:48.672979 19000 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790394221564982-779-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790394221564982-779-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790394221564982-779-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:48:48.689579 19000 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:48:48.694085 19000 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:48:48.694325 19000 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:48:48.713203 19000 rpc_server.cc:307] RPC server started. Bound to: 127.0.194.254:45107
I20260926 03:48:48.713222 19066 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.194.254:45107 every 8 connection(s)
I20260926 03:48:48.714607 19000 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790394221564982-779-0/minicluster-data/master-0/data/info.pb
I20260926 03:48:48.715096 779 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu as pid 19000
I20260926 03:48:48.715284 779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790394221564982-779-0/minicluster-data/master-0/wal/instance
I20260926 03:48:48.715937 779 external_mini_cluster.cc:1428] Setting key b9717e0588f374e4588716efe9115cd8
I20260926 03:48:48.719358 19067 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 26 03:48:48 dist-test-slave-n2th krb5kdc[18987](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394528, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.194.254@KRBTEST.COM
I20260926 03:48:48.732957 19067 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46d29631621547398f8f7e5451d9f243: Bootstrap starting.
I20260926 03:48:48.737426 19067 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 46d29631621547398f8f7e5451d9f243: Neither blocks nor log segments found. Creating new log.
I20260926 03:48:48.738713 19067 log.cc:824] T 00000000000000000000000000000000 P 46d29631621547398f8f7e5451d9f243: Log is configured to *not* fsync() on all Append() calls
I20260926 03:48:48.741899 19070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:48.717876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34388 (local address 127.0.194.254:45107)
0926 03:48:48.718698 (+ 822us) server_negotiation.cc:206] Beginning negotiation
0926 03:48:48.718716 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0926 03:48:48.718798 (+ 82us) server_negotiation.cc:407] Connection header received
0926 03:48:48.721215 (+ 2417us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:48:48.721246 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:48:48.721799 (+ 553us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:48:48.722294 (+ 495us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:48:48.723828 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:48:48.725971 (+ 2143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:48:48.727483 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:48:48.728530 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:48:48.728727 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:48:48.733104 (+ 4377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:48:48.733116 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:48:48.733137 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:48:48.733191 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:48:48.737078 (+ 3887us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:48:48.737900 (+ 822us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:48:48.737909 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:48:48.737917 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:48:48.738138 (+ 221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:48:48.738543 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:48:48.738548 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:48:48.738551 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:48:48.739322 (+ 771us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:48:48.739609 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0926 03:48:48.740094 (+ 485us) server_negotiation.cc:299] Negotiation successful
0926 03:48:48.740540 (+ 446us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":568,"thread_start_us":338,"threads_started":1}
I20260926 03:48:48.742293 19067 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46d29631621547398f8f7e5451d9f243: No bootstrap required, opened a new log
I20260926 03:48:48.750310 19067 raft_consensus.cc:348] T 00000000000000000000000000000000 P 46d29631621547398f8f7e5451d9f243 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46d29631621547398f8f7e5451d9f243" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 45107 } }
I20260926 03:48:48.750703 19067 raft_consensus.cc:374] T 00000000000000000000000000000000 P 46d29631621547398f8f7e5451d9f243 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:48:48.750835 19067 raft_consensus.cc:729] T 00000000000000000000000000000000 P 46d29631621547398f8f7e5451d9f243 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46d29631621547398f8f7e5451d9f243, State: Initialized, Role: FOLLOWER
I20260926 03:48:48.751395 19067 consensus_queue.cc:260] T 00000000000000000000000000000000 P 46d29631621547398f8f7e5451d9f243 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46d29631621547398f8f7e5451d9f243" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 45107 } }
I20260926 03:48:48.751600 19067 raft_consensus.cc:388] T 00000000000000000000000000000000 P 46d29631621547398f8f7e5451d9f243 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:48:48.751698 19067 raft_consensus.cc:482] T 00000000000000000000000000000000 P 46d29631621547398f8f7e5451d9f243 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:48:48.751819 19067 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 46d29631621547398f8f7e5451d9f243 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:48:48.753999 19067 raft_consensus.cc:504] T 00000000000000000000000000000000 P 46d29631621547398f8f7e5451d9f243 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46d29631621547398f8f7e5451d9f243" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 45107 } }
I20260926 03:48:48.754462 19067 leader_election.cc:302] T 00000000000000000000000000000000 P 46d29631621547398f8f7e5451d9f243 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 46d29631621547398f8f7e5451d9f243; no voters:
I20260926 03:48:48.755054 19067 leader_election.cc:288] T 00000000000000000000000000000000 P 46d29631621547398f8f7e5451d9f243 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:48:48.755203 19072 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 46d29631621547398f8f7e5451d9f243 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:48:48.756327 19072 raft_consensus.cc:686] T 00000000000000000000000000000000 P 46d29631621547398f8f7e5451d9f243 [term 1 LEADER]: Becoming Leader. State: Replica: 46d29631621547398f8f7e5451d9f243, State: Running, Role: LEADER
I20260926 03:48:48.756886 19072 consensus_queue.cc:237] T 00000000000000000000000000000000 P 46d29631621547398f8f7e5451d9f243 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46d29631621547398f8f7e5451d9f243" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 45107 } }
I20260926 03:48:48.757368 19067 sys_catalog.cc:564] T 00000000000000000000000000000000 P 46d29631621547398f8f7e5451d9f243 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:48:48.759989 19074 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46d29631621547398f8f7e5451d9f243 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 46d29631621547398f8f7e5451d9f243. Latest consensus state: current_term: 1 leader_uuid: "46d29631621547398f8f7e5451d9f243" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46d29631621547398f8f7e5451d9f243" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 45107 } } }
I20260926 03:48:48.760461 19074 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46d29631621547398f8f7e5451d9f243 [sys.catalog]: This master's current role is: LEADER
I20260926 03:48:48.759991 19073 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46d29631621547398f8f7e5451d9f243 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "46d29631621547398f8f7e5451d9f243" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46d29631621547398f8f7e5451d9f243" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 45107 } } }
I20260926 03:48:48.761226 19073 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46d29631621547398f8f7e5451d9f243 [sys.catalog]: This master's current role is: LEADER
I20260926 03:48:48.762936 19077 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:48:48.769078 19077 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:48:48.780962 19077 catalog_manager.cc:1347] Generated new cluster ID: 08fe80a6318a46949ab18b820f0c3648
I20260926 03:48:48.781116 19077 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:48:48.803660 19077 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:48:48.805037 19077 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:48:48.820056 19077 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 46d29631621547398f8f7e5451d9f243: Generated new TSK 0
I20260926 03:48:48.820943 19077 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.194.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.194.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.194.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790394221564982-779-0/krb5kdc/kudu_127.0.194.193.keytab.
Entry for principal kudu/127.0.194.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790394221564982-779-0/krb5kdc/kudu_127.0.194.193.keytab.
I20260926 03:48:48.878041 779 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
/tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790394221564982-779-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790394221564982-779-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790394221564982-779-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790394221564982-779-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.194.193:0
--local_ip_for_outbound_sockets=127.0.194.193
--webserver_interface=127.0.194.193
--webserver_port=0
--tserver_master_addrs=127.0.194.254:45107
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790394221564982-779-0/krb5kdc/kudu_127.0.194.193.keytab
--principal=kudu/127.0.194.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.194.212:41949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790394221564982-779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790394221564982-779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790394221564982-779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:48:49.215595 19093 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:48:49.215945 19093 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:48:49.216120 19093 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:48:49.223551 19093 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:48:49.223723 19093 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:48:49.223917 19093 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.194.193
I20260926 03:48:49.233944 19093 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.194.212:41949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790394221564982-779-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790394221564982-779-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790394221564982-779-0/krb5kdc/kudu_127.0.194.193.keytab
--principal=kudu/127.0.194.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.194.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790394221564982-779-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.194.193
--webserver_port=0
--tserver_master_addrs=127.0.194.254:45107
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.194.193
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790394221564982-779-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:48:49.235347 19093 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:48:49.236704 19093 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:48:49.249986 19098 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:48:49.249926 19101 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:48:49.250059 19099 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:48:49.250890 19093 server_base.cc:1034] running on GCE node
I20260926 03:48:49.251570 19093 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:48:49.252532 19093 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:48:49.253736 19093 hybrid_clock.cc:648] HybridClock initialized: now 1790394529253691 us; error 38 us; skew 500 ppm
I20260926 03:48:49.254069 19093 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:48:49 dist-test-slave-n2th krb5kdc[18987](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394529, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:48:49.257403 19093 init.cc:377] Logged in from keytab as kudu/127.0.194.193@KRBTEST.COM (short username kudu)
I20260926 03:48:49.258973 19093 webserver.cc:460] Webserver started at http://127.0.194.193:39101/ using document root <none> and password file <none>
I20260926 03:48:49.259681 19093 fs_manager.cc:359] Metadata directory not provided
I20260926 03:48:49.259829 19093 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:48:49.260149 19093 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:48:49.263182 19093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790394221564982-779-0/minicluster-data/ts-0/data/instance:
uuid: "916b0426be81457ca54b37874c6b4612"
format_stamp: "Formatted at 2026-09-26 03:48:49 on dist-test-slave-n2th"
server_key: "440ef7144355f1ef8f974a160188024f"
server_key_iv: "1f4231c2322f06e2c80d3c18e977fc1b"
server_key_version: "encryptionkey@0"
I20260926 03:48:49.264000 19093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790394221564982-779-0/minicluster-data/ts-0/wal/instance:
uuid: "916b0426be81457ca54b37874c6b4612"
format_stamp: "Formatted at 2026-09-26 03:48:49 on dist-test-slave-n2th"
server_key: "440ef7144355f1ef8f974a160188024f"
server_key_iv: "1f4231c2322f06e2c80d3c18e977fc1b"
server_key_version: "encryptionkey@0"
I20260926 03:48:49.269358 19093 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 03:48:49.273088 19108 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:48:49.274448 19093 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 03:48:49.274641 19093 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790394221564982-779-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790394221564982-779-0/minicluster-data/ts-0/wal
uuid: "916b0426be81457ca54b37874c6b4612"
format_stamp: "Formatted at 2026-09-26 03:48:49 on dist-test-slave-n2th"
server_key: "440ef7144355f1ef8f974a160188024f"
server_key_iv: "1f4231c2322f06e2c80d3c18e977fc1b"
server_key_version: "encryptionkey@0"
I20260926 03:48:49.274892 19093 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790394221564982-779-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790394221564982-779-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790394221564982-779-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:48:49.294468 19093 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:48:49.299206 19093 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:48:49.299532 19093 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:48:49.300550 19093 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:48:49.301949 19093 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:48:49.302052 19093 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:48:49.302204 19093 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:48:49.302273 19093 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:48:49.335671 19093 rpc_server.cc:307] RPC server started. Bound to: 127.0.194.193:40699
I20260926 03:48:49.335711 19220 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.194.193:40699 every 8 connection(s)
I20260926 03:48:49.336992 19093 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790394221564982-779-0/minicluster-data/ts-0/data/info.pb
I20260926 03:48:49.343219 779 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu as pid 19093
I20260926 03:48:49.343407 779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790394221564982-779-0/minicluster-data/ts-0/wal/instance
I20260926 03:48:49.343858 779 external_mini_cluster.cc:1428] Setting key 6e24dd3e697fdbc5a5bd603c2ba22865
Sep 26 03:48:49 dist-test-slave-n2th krb5kdc[18987](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394529, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.193@KRBTEST.COM for kudu/127.0.194.254@KRBTEST.COM
I20260926 03:48:49.357658 19223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:49.339560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.193:49769 (local address 127.0.194.254:45107)
0926 03:48:49.339991 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0926 03:48:49.339996 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:48:49.341048 (+ 1052us) server_negotiation.cc:407] Connection header received
0926 03:48:49.342946 (+ 1898us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:48:49.342951 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:48:49.343054 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:48:49.343177 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:48:49.344852 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:48:49.345588 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:48:49.347270 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:48:49.347678 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:48:49.347820 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:48:49.351609 (+ 3789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:48:49.351614 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:48:49.351617 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:48:49.351651 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:48:49.354380 (+ 2729us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:48:49.355234 (+ 854us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:48:49.355245 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:48:49.355248 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:48:49.355368 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:48:49.355809 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:48:49.355815 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:48:49.355822 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:48:49.356112 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:48:49.356274 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0926 03:48:49.357013 (+ 739us) server_negotiation.cc:299] Negotiation successful
0926 03:48:49.357293 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":207,"threads_started":1}
I20260926 03:48:49.358713 19224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:49.340162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:45107 (local address 127.0.194.193:49769)
0926 03:48:49.340862 (+ 700us) negotiation.cc:107] Waiting for socket to connect
0926 03:48:49.340912 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 03:48:49.342624 (+ 1712us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:48:49.343456 (+ 832us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:48:49.343485 (+ 29us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:48:49.344103 (+ 618us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:48:49.344675 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:48:49.344693 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:48:49.345747 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:48:49.345752 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:48:49.347125 (+ 1373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:48:49.347137 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:48:49.347920 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:48:49.347928 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:48:49.348170 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:48:49.348874 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:48:49.348902 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:48:49.351368 (+ 2466us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:48:49.354539 (+ 3171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:48:49.354548 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:48:49.354566 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:48:49.355100 (+ 534us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:48:49.355487 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:48:49.355491 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:48:49.355495 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:48:49.355688 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:48:49.356241 (+ 553us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:48:49.356250 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:48:49.356623 (+ 373us) client_negotiation.cc:769] Sending connection context
0926 03:48:49.356899 (+ 276us) client_negotiation.cc:240] Negotiation successful
0926 03:48:49.357486 (+ 587us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":280,"threads_started":1}
I20260926 03:48:49.360373 19221 heartbeater.cc:347] Connected to a master server at 127.0.194.254:45107
I20260926 03:48:49.360710 19221 heartbeater.cc:464] Registering TS with master...
I20260926 03:48:49.361608 19221 heartbeater.cc:511] Master 127.0.194.254:45107 requested a full tablet report, sending...
I20260926 03:48:49.364148 19032 ts_manager.cc:194] Registered new tserver with Master: 916b0426be81457ca54b37874c6b4612 (127.0.194.193:40699)
I20260926 03:48:49.366457 19032 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.194.193@KRBTEST.COM'} at 127.0.194.193:49769
WARNING: no policy specified for kudu/127.0.194.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.194.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.194.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790394221564982-779-0/krb5kdc/kudu_127.0.194.194.keytab.
Entry for principal kudu/127.0.194.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790394221564982-779-0/krb5kdc/kudu_127.0.194.194.keytab.
I20260926 03:48:49.399348 779 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
/tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790394221564982-779-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790394221564982-779-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790394221564982-779-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790394221564982-779-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.194.194:0
--local_ip_for_outbound_sockets=127.0.194.194
--webserver_interface=127.0.194.194
--webserver_port=0
--tserver_master_addrs=127.0.194.254:45107
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790394221564982-779-0/krb5kdc/kudu_127.0.194.194.keytab
--principal=kudu/127.0.194.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.194.212:41949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790394221564982-779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790394221564982-779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790394221564982-779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:48:49.738205 19227 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:48:49.738487 19227 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:48:49.738631 19227 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:48:49.746085 19227 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:48:49.746209 19227 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:48:49.746404 19227 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.194.194
I20260926 03:48:49.756448 19227 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.194.212:41949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790394221564982-779-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790394221564982-779-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790394221564982-779-0/krb5kdc/kudu_127.0.194.194.keytab
--principal=kudu/127.0.194.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.194.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790394221564982-779-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.194.194
--webserver_port=0
--tserver_master_addrs=127.0.194.254:45107
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.194.194
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790394221564982-779-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:48:49.757893 19227 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:48:49.759279 19227 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:48:49.772933 19232 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:48:49.772889 19235 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:48:49.772971 19233 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:48:49.774151 19227 server_base.cc:1034] running on GCE node
I20260926 03:48:49.774842 19227 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:48:49.775810 19227 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:48:49.776996 19227 hybrid_clock.cc:648] HybridClock initialized: now 1790394529776944 us; error 39 us; skew 500 ppm
I20260926 03:48:49.777316 19227 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:48:49 dist-test-slave-n2th krb5kdc[18987](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394529, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:48:49.780625 19227 init.cc:377] Logged in from keytab as kudu/127.0.194.194@KRBTEST.COM (short username kudu)
I20260926 03:48:49.782186 19227 webserver.cc:460] Webserver started at http://127.0.194.194:42713/ using document root <none> and password file <none>
I20260926 03:48:49.782953 19227 fs_manager.cc:359] Metadata directory not provided
I20260926 03:48:49.783092 19227 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:48:49.783407 19227 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:48:49.786341 19227 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790394221564982-779-0/minicluster-data/ts-1/data/instance:
uuid: "bb322be6e99448b3afae9e92cfcf466f"
format_stamp: "Formatted at 2026-09-26 03:48:49 on dist-test-slave-n2th"
server_key: "c694fb3f5a70d504861793a982bc268b"
server_key_iv: "2c3cb24f7e663db858511e46deeb516c"
server_key_version: "encryptionkey@0"
I20260926 03:48:49.787211 19227 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790394221564982-779-0/minicluster-data/ts-1/wal/instance:
uuid: "bb322be6e99448b3afae9e92cfcf466f"
format_stamp: "Formatted at 2026-09-26 03:48:49 on dist-test-slave-n2th"
server_key: "c694fb3f5a70d504861793a982bc268b"
server_key_iv: "2c3cb24f7e663db858511e46deeb516c"
server_key_version: "encryptionkey@0"
I20260926 03:48:49.792428 19227 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 03:48:49.796056 19242 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:48:49.797385 19227 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 03:48:49.797580 19227 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790394221564982-779-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790394221564982-779-0/minicluster-data/ts-1/wal
uuid: "bb322be6e99448b3afae9e92cfcf466f"
format_stamp: "Formatted at 2026-09-26 03:48:49 on dist-test-slave-n2th"
server_key: "c694fb3f5a70d504861793a982bc268b"
server_key_iv: "2c3cb24f7e663db858511e46deeb516c"
server_key_version: "encryptionkey@0"
I20260926 03:48:49.797811 19227 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790394221564982-779-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790394221564982-779-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790394221564982-779-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:48:49.819700 19227 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:48:49.824180 19227 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:48:49.824483 19227 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:48:49.825465 19227 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:48:49.826937 19227 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:48:49.827045 19227 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:48:49.827181 19227 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:48:49.827248 19227 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:48:49.861881 19227 rpc_server.cc:307] RPC server started. Bound to: 127.0.194.194:33525
I20260926 03:48:49.861925 19354 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.194.194:33525 every 8 connection(s)
I20260926 03:48:49.863327 19227 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790394221564982-779-0/minicluster-data/ts-1/data/info.pb
I20260926 03:48:49.864820 779 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu as pid 19227
I20260926 03:48:49.865007 779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790394221564982-779-0/minicluster-data/ts-1/wal/instance
I20260926 03:48:49.865445 779 external_mini_cluster.cc:1428] Setting key ecbed115705aff2eac3db983a8960ca1
Sep 26 03:48:49 dist-test-slave-n2th krb5kdc[18987](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394529, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.194@KRBTEST.COM for kudu/127.0.194.254@KRBTEST.COM
I20260926 03:48:49.887533 19357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:49.866091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.194:49373 (local address 127.0.194.254:45107)
0926 03:48:49.866550 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0926 03:48:49.866555 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:48:49.868198 (+ 1643us) server_negotiation.cc:407] Connection header received
0926 03:48:49.870589 (+ 2391us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:48:49.870598 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:48:49.870694 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:48:49.870851 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:48:49.872597 (+ 1746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:48:49.873339 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:48:49.875249 (+ 1910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:48:49.875699 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:48:49.875837 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:48:49.880973 (+ 5136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:48:49.880982 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:48:49.880990 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:48:49.881035 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:48:49.883773 (+ 2738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:48:49.884889 (+ 1116us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:48:49.884893 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:48:49.884895 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:48:49.885030 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:48:49.885573 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:48:49.885576 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:48:49.885578 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:48:49.885856 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:48:49.885993 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0926 03:48:49.887045 (+ 1052us) server_negotiation.cc:299] Negotiation successful
0926 03:48:49.887243 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":234,"threads_started":1}
I20260926 03:48:49.889127 19358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:49.866848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:45107 (local address 127.0.194.194:49373)
0926 03:48:49.867831 (+ 983us) negotiation.cc:107] Waiting for socket to connect
0926 03:48:49.867896 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0926 03:48:49.870259 (+ 2363us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:48:49.871121 (+ 862us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:48:49.871131 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:48:49.871713 (+ 582us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:48:49.872327 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:48:49.872345 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:48:49.873519 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:48:49.873525 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:48:49.875092 (+ 1567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:48:49.875105 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:48:49.875964 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:48:49.875974 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:48:49.876306 (+ 332us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:48:49.877149 (+ 843us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:48:49.877195 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:48:49.880600 (+ 3405us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:48:49.883970 (+ 3370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:48:49.883984 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:48:49.884008 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:48:49.884745 (+ 737us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:48:49.885150 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:48:49.885158 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:48:49.885164 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:48:49.885456 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:48:49.885990 (+ 534us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:48:49.886003 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:48:49.886533 (+ 530us) client_negotiation.cc:769] Sending connection context
0926 03:48:49.886886 (+ 353us) client_negotiation.cc:240] Negotiation successful
0926 03:48:49.887545 (+ 659us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":638,"thread_start_us":339,"threads_started":1}
I20260926 03:48:49.890797 19355 heartbeater.cc:347] Connected to a master server at 127.0.194.254:45107
I20260926 03:48:49.891129 19355 heartbeater.cc:464] Registering TS with master...
I20260926 03:48:49.891893 19355 heartbeater.cc:511] Master 127.0.194.254:45107 requested a full tablet report, sending...
I20260926 03:48:49.893409 19032 ts_manager.cc:194] Registered new tserver with Master: bb322be6e99448b3afae9e92cfcf466f (127.0.194.194:33525)
I20260926 03:48:49.894225 19032 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.194.194@KRBTEST.COM'} at 127.0.194.194:49373
WARNING: no policy specified for kudu/127.0.194.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.194.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.194.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790394221564982-779-0/krb5kdc/kudu_127.0.194.195.keytab.
Entry for principal kudu/127.0.194.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790394221564982-779-0/krb5kdc/kudu_127.0.194.195.keytab.
I20260926 03:48:49.924372 779 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
/tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790394221564982-779-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790394221564982-779-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790394221564982-779-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790394221564982-779-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.194.195:0
--local_ip_for_outbound_sockets=127.0.194.195
--webserver_interface=127.0.194.195
--webserver_port=0
--tserver_master_addrs=127.0.194.254:45107
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790394221564982-779-0/krb5kdc/kudu_127.0.194.195.keytab
--principal=kudu/127.0.194.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.194.212:41949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790394221564982-779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790394221564982-779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790394221564982-779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:48:50.260843 19361 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:48:50.261178 19361 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:48:50.261336 19361 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:48:50.268963 19361 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:48:50.269117 19361 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:48:50.269290 19361 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.194.195
I20260926 03:48:50.280141 19361 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.194.212:41949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790394221564982-779-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790394221564982-779-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790394221564982-779-0/krb5kdc/kudu_127.0.194.195.keytab
--principal=kudu/127.0.194.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.194.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790394221564982-779-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.194.195
--webserver_port=0
--tserver_master_addrs=127.0.194.254:45107
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.194.195
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790394221564982-779-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:48:50.281515 19361 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:48:50.283130 19361 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:48:50.296829 19367 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:48:50.296888 19369 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:48:50.296957 19366 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:48:50.297892 19361 server_base.cc:1034] running on GCE node
I20260926 03:48:50.298522 19361 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:48:50.299568 19361 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:48:50.300799 19361 hybrid_clock.cc:648] HybridClock initialized: now 1790394530300754 us; error 43 us; skew 500 ppm
I20260926 03:48:50.301167 19361 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:48:50 dist-test-slave-n2th krb5kdc[18987](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394530, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:48:50.304548 19361 init.cc:377] Logged in from keytab as kudu/127.0.194.195@KRBTEST.COM (short username kudu)
I20260926 03:48:50.306072 19361 webserver.cc:460] Webserver started at http://127.0.194.195:44227/ using document root <none> and password file <none>
I20260926 03:48:50.306869 19361 fs_manager.cc:359] Metadata directory not provided
I20260926 03:48:50.307020 19361 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:48:50.307341 19361 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:48:50.310339 19361 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790394221564982-779-0/minicluster-data/ts-2/data/instance:
uuid: "ee200939409f42beb9d62636bb226e77"
format_stamp: "Formatted at 2026-09-26 03:48:50 on dist-test-slave-n2th"
server_key: "77e86784f5311a675e2efb6dc7cdb66b"
server_key_iv: "819ac263f5282c92dee298327ead5c66"
server_key_version: "encryptionkey@0"
I20260926 03:48:50.311179 19361 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790394221564982-779-0/minicluster-data/ts-2/wal/instance:
uuid: "ee200939409f42beb9d62636bb226e77"
format_stamp: "Formatted at 2026-09-26 03:48:50 on dist-test-slave-n2th"
server_key: "77e86784f5311a675e2efb6dc7cdb66b"
server_key_iv: "819ac263f5282c92dee298327ead5c66"
server_key_version: "encryptionkey@0"
I20260926 03:48:50.316500 19361 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 03:48:50.320138 19376 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:48:50.321530 19361 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 03:48:50.321712 19361 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790394221564982-779-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790394221564982-779-0/minicluster-data/ts-2/wal
uuid: "ee200939409f42beb9d62636bb226e77"
format_stamp: "Formatted at 2026-09-26 03:48:50 on dist-test-slave-n2th"
server_key: "77e86784f5311a675e2efb6dc7cdb66b"
server_key_iv: "819ac263f5282c92dee298327ead5c66"
server_key_version: "encryptionkey@0"
I20260926 03:48:50.321944 19361 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790394221564982-779-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790394221564982-779-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790394221564982-779-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:48:50.335000 19361 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:48:50.340325 19361 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:48:50.340669 19361 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:48:50.341862 19361 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:48:50.343293 19361 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:48:50.343366 19361 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:48:50.343462 19361 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:48:50.343509 19361 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:48:50.370393 19221 heartbeater.cc:503] Master 127.0.194.254:45107 was elected leader, sending a full tablet report...
I20260926 03:48:50.376292 19361 rpc_server.cc:307] RPC server started. Bound to: 127.0.194.195:35147
I20260926 03:48:50.376341 19488 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.194.195:35147 every 8 connection(s)
I20260926 03:48:50.377646 19361 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790394221564982-779-0/minicluster-data/ts-2/data/info.pb
I20260926 03:48:50.379230 779 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu as pid 19361
I20260926 03:48:50.379443 779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790394221564982-779-0/minicluster-data/ts-2/wal/instance
I20260926 03:48:50.379925 779 external_mini_cluster.cc:1428] Setting key 5dc24daedf1b304d7404d147ede79c41
Sep 26 03:48:50 dist-test-slave-n2th krb5kdc[18987](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394530, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.195@KRBTEST.COM for kudu/127.0.194.254@KRBTEST.COM
I20260926 03:48:50.399307 19357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:50.380427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.195:48457 (local address 127.0.194.254:45107)
0926 03:48:50.380616 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0926 03:48:50.380623 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:48:50.382013 (+ 1390us) server_negotiation.cc:407] Connection header received
0926 03:48:50.384058 (+ 2045us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:48:50.384063 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:48:50.384149 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:48:50.384260 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:48:50.385781 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:48:50.386520 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:48:50.388116 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:48:50.388532 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:48:50.388664 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:48:50.392547 (+ 3883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:48:50.392553 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:48:50.392556 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:48:50.392595 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:48:50.396218 (+ 3623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:48:50.397103 (+ 885us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:48:50.397107 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:48:50.397109 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:48:50.397230 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:48:50.397637 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:48:50.397640 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:48:50.397642 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:48:50.397916 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:48:50.398044 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0926 03:48:50.398868 (+ 824us) server_negotiation.cc:299] Negotiation successful
0926 03:48:50.399072 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 03:48:50.400355 19491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:50.380954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:45107 (local address 127.0.194.195:48457)
0926 03:48:50.381859 (+ 905us) negotiation.cc:107] Waiting for socket to connect
0926 03:48:50.381907 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 03:48:50.383802 (+ 1895us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:48:50.384507 (+ 705us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:48:50.384516 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:48:50.385047 (+ 531us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:48:50.385620 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:48:50.385637 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:48:50.386657 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:48:50.386662 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:48:50.387972 (+ 1310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:48:50.387985 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:48:50.388756 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:48:50.388764 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:48:50.389026 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:48:50.389817 (+ 791us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:48:50.389839 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:48:50.392265 (+ 2426us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:48:50.396397 (+ 4132us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:48:50.396405 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:48:50.396422 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:48:50.396984 (+ 562us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:48:50.397322 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:48:50.397325 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:48:50.397327 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:48:50.397541 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:48:50.398035 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:48:50.398042 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:48:50.398479 (+ 437us) client_negotiation.cc:769] Sending connection context
0926 03:48:50.398724 (+ 245us) client_negotiation.cc:240] Negotiation successful
0926 03:48:50.399231 (+ 507us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":604,"thread_start_us":335,"threads_started":1}
I20260926 03:48:50.401836 19489 heartbeater.cc:347] Connected to a master server at 127.0.194.254:45107
I20260926 03:48:50.402174 19489 heartbeater.cc:464] Registering TS with master...
I20260926 03:48:50.402997 19489 heartbeater.cc:511] Master 127.0.194.254:45107 requested a full tablet report, sending...
I20260926 03:48:50.404428 19032 ts_manager.cc:194] Registered new tserver with Master: ee200939409f42beb9d62636bb226e77 (127.0.194.195:35147)
I20260926 03:48:50.405361 19032 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.194.195@KRBTEST.COM'} at 127.0.194.195:48457
I20260926 03:48:50.408381 779 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 03:48:50.425194 19357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:50.412448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34400 (local address 127.0.194.254:45107)
0926 03:48:50.412642 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0926 03:48:50.412647 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:48:50.413109 (+ 462us) server_negotiation.cc:407] Connection header received
0926 03:48:50.413251 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:48:50.413255 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:48:50.413330 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:48:50.413422 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:48:50.414881 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:48:50.415587 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:48:50.416862 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:48:50.417252 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:48:50.417348 (+ 96us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:48:50.419899 (+ 2551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:48:50.419904 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:48:50.419907 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:48:50.419957 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:48:50.422513 (+ 2556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:48:50.423440 (+ 927us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:48:50.423444 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:48:50.423446 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:48:50.423575 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:48:50.423928 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:48:50.423931 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:48:50.423933 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:48:50.424226 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:48:50.424432 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0926 03:48:50.424735 (+ 303us) server_negotiation.cc:299] Negotiation successful
0926 03:48:50.424867 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260926 03:48:50.429500 19032 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34400:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 03:48:50.436599 19032 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 03:48:50.463191 19503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:50.449317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.195:35147 (local address 127.0.0.1:53102)
0926 03:48:50.451578 (+ 2261us) negotiation.cc:107] Waiting for socket to connect
0926 03:48:50.451598 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 03:48:50.451782 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:48:50.452918 (+ 1136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:48:50.452926 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:48:50.452968 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:48:50.453566 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:48:50.453581 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:48:50.456246 (+ 2665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:48:50.456254 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:48:50.460194 (+ 3940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:48:50.460209 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:48:50.462066 (+ 1857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:48:50.462072 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:48:50.462315 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:48:50.462376 (+ 61us) client_negotiation.cc:769] Sending connection context
0926 03:48:50.462516 (+ 140us) client_negotiation.cc:240] Negotiation successful
0926 03:48:50.462717 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2144,"mutex_wait_us":1,"thread_start_us":2021,"threads_started":1}
I20260926 03:48:50.464112 19504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:50.449886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53102 (local address 127.0.194.195:35147)
0926 03:48:50.451122 (+ 1236us) server_negotiation.cc:206] Beginning negotiation
0926 03:48:50.451132 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 03:48:50.451682 (+ 550us) server_negotiation.cc:407] Connection header received
0926 03:48:50.451989 (+ 307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:48:50.451999 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:48:50.452317 (+ 318us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:48:50.452512 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:48:50.453803 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:48:50.456016 (+ 2213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:48:50.460360 (+ 4344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:48:50.461682 (+ 1322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:48:50.461825 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:48:50.462090 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0926 03:48:50.463597 (+ 1507us) server_negotiation.cc:299] Negotiation successful
0926 03:48:50.463777 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1035,"thread_start_us":886,"threads_started":1}
I20260926 03:48:50.464785 19502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:50.449181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40026 (local address 127.0.194.194:33525)
0926 03:48:50.452059 (+ 2878us) server_negotiation.cc:206] Beginning negotiation
0926 03:48:50.452071 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 03:48:50.452102 (+ 31us) server_negotiation.cc:407] Connection header received
0926 03:48:50.452203 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:48:50.452214 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:48:50.452452 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:48:50.455352 (+ 2900us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:48:50.455665 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:48:50.459308 (+ 3643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:48:50.462073 (+ 2765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:48:50.463585 (+ 1512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:48:50.463719 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:48:50.463946 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0926 03:48:50.464434 (+ 488us) server_negotiation.cc:299] Negotiation successful
0926 03:48:50.464560 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2693,"thread_start_us":2579,"threads_started":1}
I20260926 03:48:50.465531 19501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:50.448999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.194:33525 (local address 127.0.0.1:40026)
0926 03:48:50.450173 (+ 1174us) negotiation.cc:107] Waiting for socket to connect
0926 03:48:50.450221 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 03:48:50.450411 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:48:50.452759 (+ 2348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:48:50.452771 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:48:50.452814 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:48:50.453471 (+ 657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:48:50.453493 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:48:50.459508 (+ 6015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:48:50.459522 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:48:50.461850 (+ 2328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:48:50.461869 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:48:50.464006 (+ 2137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:48:50.464011 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:48:50.464138 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:48:50.464163 (+ 25us) client_negotiation.cc:769] Sending connection context
0926 03:48:50.464295 (+ 132us) client_negotiation.cc:240] Negotiation successful
0926 03:48:50.465142 (+ 847us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":901,"mutex_wait_us":744,"thread_start_us":759,"threads_started":1}
I20260926 03:48:50.468811 19505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:50.451748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.193:40699 (local address 127.0.0.1:55056)
0926 03:48:50.460844 (+ 9096us) negotiation.cc:107] Waiting for socket to connect
0926 03:48:50.460863 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 03:48:50.461007 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:48:50.461682 (+ 675us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:48:50.461687 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:48:50.461717 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:48:50.462160 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:48:50.462171 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:48:50.464097 (+ 1926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:48:50.464104 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:48:50.465925 (+ 1821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:48:50.465938 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:48:50.467994 (+ 2056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:48:50.468003 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:48:50.468180 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:48:50.468215 (+ 35us) client_negotiation.cc:769] Sending connection context
0926 03:48:50.468306 (+ 91us) client_negotiation.cc:240] Negotiation successful
0926 03:48:50.468439 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8955,"mutex_wait_us":3,"thread_start_us":8786,"threads_started":1}
I20260926 03:48:50.468329 19424 tablet_service.cc:1467] Processing CreateTablet for tablet e1b536ee9f0e4c6da21bc1c9781e400e (DEFAULT_TABLE table=test-table [id=85f920ff16ff43968bd07dbe020d622c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:48:50.469238 19506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:50.452307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55056 (local address 127.0.194.193:40699)
0926 03:48:50.460142 (+ 7835us) server_negotiation.cc:206] Beginning negotiation
0926 03:48:50.460149 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:48:50.460990 (+ 841us) server_negotiation.cc:407] Connection header received
0926 03:48:50.461161 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:48:50.461169 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:48:50.461462 (+ 293us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:48:50.461666 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:48:50.462326 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:48:50.463919 (+ 1593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:48:50.466083 (+ 2164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:48:50.467461 (+ 1378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:48:50.467614 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:48:50.467834 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0926 03:48:50.468724 (+ 890us) server_negotiation.cc:299] Negotiation successful
0926 03:48:50.468901 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7662,"thread_start_us":7493,"threads_started":1}
I20260926 03:48:50.469379 19290 tablet_service.cc:1467] Processing CreateTablet for tablet e1b536ee9f0e4c6da21bc1c9781e400e (DEFAULT_TABLE table=test-table [id=85f920ff16ff43968bd07dbe020d622c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:48:50.470026 19424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1b536ee9f0e4c6da21bc1c9781e400e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:48:50.470687 19290 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1b536ee9f0e4c6da21bc1c9781e400e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:48:50.472879 19156 tablet_service.cc:1467] Processing CreateTablet for tablet e1b536ee9f0e4c6da21bc1c9781e400e (DEFAULT_TABLE table=test-table [id=85f920ff16ff43968bd07dbe020d622c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:48:50.474234 19156 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1b536ee9f0e4c6da21bc1c9781e400e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:48:50.485421 19508 tablet_bootstrap.cc:492] T e1b536ee9f0e4c6da21bc1c9781e400e P bb322be6e99448b3afae9e92cfcf466f: Bootstrap starting.
I20260926 03:48:50.487371 19507 tablet_bootstrap.cc:492] T e1b536ee9f0e4c6da21bc1c9781e400e P ee200939409f42beb9d62636bb226e77: Bootstrap starting.
I20260926 03:48:50.489267 19509 tablet_bootstrap.cc:492] T e1b536ee9f0e4c6da21bc1c9781e400e P 916b0426be81457ca54b37874c6b4612: Bootstrap starting.
I20260926 03:48:50.489444 19508 tablet_bootstrap.cc:654] T e1b536ee9f0e4c6da21bc1c9781e400e P bb322be6e99448b3afae9e92cfcf466f: Neither blocks nor log segments found. Creating new log.
I20260926 03:48:50.491372 19508 log.cc:824] T e1b536ee9f0e4c6da21bc1c9781e400e P bb322be6e99448b3afae9e92cfcf466f: Log is configured to *not* fsync() on all Append() calls
I20260926 03:48:50.491778 19507 tablet_bootstrap.cc:654] T e1b536ee9f0e4c6da21bc1c9781e400e P ee200939409f42beb9d62636bb226e77: Neither blocks nor log segments found. Creating new log.
I20260926 03:48:50.492769 19507 log.cc:824] T e1b536ee9f0e4c6da21bc1c9781e400e P ee200939409f42beb9d62636bb226e77: Log is configured to *not* fsync() on all Append() calls
I20260926 03:48:50.493137 19509 tablet_bootstrap.cc:654] T e1b536ee9f0e4c6da21bc1c9781e400e P 916b0426be81457ca54b37874c6b4612: Neither blocks nor log segments found. Creating new log.
I20260926 03:48:50.494367 19509 log.cc:824] T e1b536ee9f0e4c6da21bc1c9781e400e P 916b0426be81457ca54b37874c6b4612: Log is configured to *not* fsync() on all Append() calls
I20260926 03:48:50.495779 19508 tablet_bootstrap.cc:492] T e1b536ee9f0e4c6da21bc1c9781e400e P bb322be6e99448b3afae9e92cfcf466f: No bootstrap required, opened a new log
I20260926 03:48:50.496138 19508 ts_tablet_manager.cc:1397] T e1b536ee9f0e4c6da21bc1c9781e400e P bb322be6e99448b3afae9e92cfcf466f: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20260926 03:48:50.496258 19507 tablet_bootstrap.cc:492] T e1b536ee9f0e4c6da21bc1c9781e400e P ee200939409f42beb9d62636bb226e77: No bootstrap required, opened a new log
I20260926 03:48:50.496524 19507 ts_tablet_manager.cc:1397] T e1b536ee9f0e4c6da21bc1c9781e400e P ee200939409f42beb9d62636bb226e77: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.006s
I20260926 03:48:50.498555 19509 tablet_bootstrap.cc:492] T e1b536ee9f0e4c6da21bc1c9781e400e P 916b0426be81457ca54b37874c6b4612: No bootstrap required, opened a new log
I20260926 03:48:50.498857 19509 ts_tablet_manager.cc:1397] T e1b536ee9f0e4c6da21bc1c9781e400e P 916b0426be81457ca54b37874c6b4612: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.004s
I20260926 03:48:50.504287 19508 raft_consensus.cc:348] T e1b536ee9f0e4c6da21bc1c9781e400e P bb322be6e99448b3afae9e92cfcf466f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb322be6e99448b3afae9e92cfcf466f" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 33525 } } peers { permanent_uuid: "ee200939409f42beb9d62636bb226e77" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 35147 } } peers { permanent_uuid: "916b0426be81457ca54b37874c6b4612" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 40699 } }
I20260926 03:48:50.504706 19508 raft_consensus.cc:374] T e1b536ee9f0e4c6da21bc1c9781e400e P bb322be6e99448b3afae9e92cfcf466f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:48:50.504840 19508 raft_consensus.cc:729] T e1b536ee9f0e4c6da21bc1c9781e400e P bb322be6e99448b3afae9e92cfcf466f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb322be6e99448b3afae9e92cfcf466f, State: Initialized, Role: FOLLOWER
I20260926 03:48:50.505496 19508 consensus_queue.cc:260] T e1b536ee9f0e4c6da21bc1c9781e400e P bb322be6e99448b3afae9e92cfcf466f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb322be6e99448b3afae9e92cfcf466f" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 33525 } } peers { permanent_uuid: "ee200939409f42beb9d62636bb226e77" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 35147 } } peers { permanent_uuid: "916b0426be81457ca54b37874c6b4612" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 40699 } }
I20260926 03:48:50.506628 19355 heartbeater.cc:503] Master 127.0.194.254:45107 was elected leader, sending a full tablet report...
I20260926 03:48:50.507486 19508 ts_tablet_manager.cc:1428] T e1b536ee9f0e4c6da21bc1c9781e400e P bb322be6e99448b3afae9e92cfcf466f: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
I20260926 03:48:50.507192 19507 raft_consensus.cc:348] T e1b536ee9f0e4c6da21bc1c9781e400e P ee200939409f42beb9d62636bb226e77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb322be6e99448b3afae9e92cfcf466f" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 33525 } } peers { permanent_uuid: "ee200939409f42beb9d62636bb226e77" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 35147 } } peers { permanent_uuid: "916b0426be81457ca54b37874c6b4612" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 40699 } }
I20260926 03:48:50.507719 19507 raft_consensus.cc:374] T e1b536ee9f0e4c6da21bc1c9781e400e P ee200939409f42beb9d62636bb226e77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:48:50.507824 19507 raft_consensus.cc:729] T e1b536ee9f0e4c6da21bc1c9781e400e P ee200939409f42beb9d62636bb226e77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee200939409f42beb9d62636bb226e77, State: Initialized, Role: FOLLOWER
I20260926 03:48:50.508584 19507 consensus_queue.cc:260] T e1b536ee9f0e4c6da21bc1c9781e400e P ee200939409f42beb9d62636bb226e77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb322be6e99448b3afae9e92cfcf466f" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 33525 } } peers { permanent_uuid: "ee200939409f42beb9d62636bb226e77" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 35147 } } peers { permanent_uuid: "916b0426be81457ca54b37874c6b4612" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 40699 } }
I20260926 03:48:50.509821 19489 heartbeater.cc:503] Master 127.0.194.254:45107 was elected leader, sending a full tablet report...
I20260926 03:48:50.509588 19509 raft_consensus.cc:348] T e1b536ee9f0e4c6da21bc1c9781e400e P 916b0426be81457ca54b37874c6b4612 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb322be6e99448b3afae9e92cfcf466f" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 33525 } } peers { permanent_uuid: "ee200939409f42beb9d62636bb226e77" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 35147 } } peers { permanent_uuid: "916b0426be81457ca54b37874c6b4612" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 40699 } }
I20260926 03:48:50.510084 19509 raft_consensus.cc:374] T e1b536ee9f0e4c6da21bc1c9781e400e P 916b0426be81457ca54b37874c6b4612 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:48:50.510193 19509 raft_consensus.cc:729] T e1b536ee9f0e4c6da21bc1c9781e400e P 916b0426be81457ca54b37874c6b4612 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 916b0426be81457ca54b37874c6b4612, State: Initialized, Role: FOLLOWER
I20260926 03:48:50.510567 19507 ts_tablet_manager.cc:1428] T e1b536ee9f0e4c6da21bc1c9781e400e P ee200939409f42beb9d62636bb226e77: Time spent starting tablet: real 0.014s user 0.015s sys 0.000s
I20260926 03:48:50.510955 19509 consensus_queue.cc:260] T e1b536ee9f0e4c6da21bc1c9781e400e P 916b0426be81457ca54b37874c6b4612 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb322be6e99448b3afae9e92cfcf466f" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 33525 } } peers { permanent_uuid: "ee200939409f42beb9d62636bb226e77" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 35147 } } peers { permanent_uuid: "916b0426be81457ca54b37874c6b4612" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 40699 } }
I20260926 03:48:50.512933 19509 ts_tablet_manager.cc:1428] T e1b536ee9f0e4c6da21bc1c9781e400e P 916b0426be81457ca54b37874c6b4612: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
W20260926 03:48:50.589399 19222 tablet.cc:2369] T e1b536ee9f0e4c6da21bc1c9781e400e P 916b0426be81457ca54b37874c6b4612: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 03:48:50.615434 19356 tablet.cc:2369] T e1b536ee9f0e4c6da21bc1c9781e400e P bb322be6e99448b3afae9e92cfcf466f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 03:48:50.629415 19490 tablet.cc:2369] T e1b536ee9f0e4c6da21bc1c9781e400e P ee200939409f42beb9d62636bb226e77: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 03:48:50.860793 19514 raft_consensus.cc:482] T e1b536ee9f0e4c6da21bc1c9781e400e P ee200939409f42beb9d62636bb226e77 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 03:48:50.861188 19514 raft_consensus.cc:504] T e1b536ee9f0e4c6da21bc1c9781e400e P ee200939409f42beb9d62636bb226e77 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb322be6e99448b3afae9e92cfcf466f" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 33525 } } peers { permanent_uuid: "ee200939409f42beb9d62636bb226e77" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 35147 } } peers { permanent_uuid: "916b0426be81457ca54b37874c6b4612" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 40699 } }
I20260926 03:48:50.863386 19514 leader_election.cc:288] T e1b536ee9f0e4c6da21bc1c9781e400e P ee200939409f42beb9d62636bb226e77 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bb322be6e99448b3afae9e92cfcf466f (127.0.194.194:33525), 916b0426be81457ca54b37874c6b4612 (127.0.194.193:40699)
I20260926 03:48:50.870857 19506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:50.864218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.195:58633 (local address 127.0.194.193:40699)
0926 03:48:50.864461 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0926 03:48:50.864469 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:48:50.864492 (+ 23us) server_negotiation.cc:407] Connection header received
0926 03:48:50.864575 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:48:50.864582 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:48:50.864660 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:48:50.864762 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:48:50.865212 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:48:50.866078 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:48:50.868587 (+ 2509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:48:50.869854 (+ 1267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:48:50.870026 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:48:50.870115 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0926 03:48:50.870412 (+ 297us) server_negotiation.cc:299] Negotiation successful
0926 03:48:50.870553 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 03:48:50.870874 19491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:50.863830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.193:40699 (local address 127.0.194.195:58633)
0926 03:48:50.864144 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0926 03:48:50.864168 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 03:48:50.864296 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:48:50.864802 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:48:50.864808 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:48:50.864844 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:48:50.865045 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:48:50.865055 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:48:50.866217 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:48:50.866245 (+ 28us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:48:50.868417 (+ 2172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:48:50.868432 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:48:50.870034 (+ 1602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:48:50.870040 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:48:50.870234 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:48:50.870270 (+ 36us) client_negotiation.cc:769] Sending connection context
0926 03:48:50.870367 (+ 97us) client_negotiation.cc:240] Negotiation successful
0926 03:48:50.870508 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"mutex_wait_us":24}
I20260926 03:48:50.871587 19502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:50.863813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.195:38885 (local address 127.0.194.194:33525)
0926 03:48:50.864034 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0926 03:48:50.864041 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:48:50.864937 (+ 896us) server_negotiation.cc:407] Connection header received
0926 03:48:50.865173 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:48:50.865178 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:48:50.865263 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:48:50.865359 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:48:50.865963 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:48:50.867074 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:48:50.869128 (+ 2054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:48:50.870389 (+ 1261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:48:50.870511 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:48:50.870576 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 03:48:50.871196 (+ 620us) server_negotiation.cc:299] Negotiation successful
0926 03:48:50.871329 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260926 03:48:50.872340 19516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:50.864083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.194:33525 (local address 127.0.194.195:38885)
0926 03:48:50.864839 (+ 756us) negotiation.cc:107] Waiting for socket to connect
0926 03:48:50.864856 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 03:48:50.865029 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:48:50.865389 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:48:50.865396 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:48:50.865423 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:48:50.865773 (+ 350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:48:50.865784 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:48:50.867217 (+ 1433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:48:50.867224 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:48:50.868938 (+ 1714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:48:50.868957 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:48:50.870799 (+ 1842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:48:50.870804 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:48:50.870994 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:48:50.871027 (+ 33us) client_negotiation.cc:769] Sending connection context
0926 03:48:50.871774 (+ 747us) client_negotiation.cc:240] Negotiation successful
0926 03:48:50.871928 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":595,"mutex_wait_us":1,"thread_start_us":414,"threads_started":1}
I20260926 03:48:50.872367 19176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1b536ee9f0e4c6da21bc1c9781e400e" candidate_uuid: "ee200939409f42beb9d62636bb226e77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "916b0426be81457ca54b37874c6b4612" is_pre_election: true
I20260926 03:48:50.872941 19176 raft_consensus.cc:2454] T e1b536ee9f0e4c6da21bc1c9781e400e P 916b0426be81457ca54b37874c6b4612 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee200939409f42beb9d62636bb226e77 in term 0.
I20260926 03:48:50.873499 19310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1b536ee9f0e4c6da21bc1c9781e400e" candidate_uuid: "ee200939409f42beb9d62636bb226e77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb322be6e99448b3afae9e92cfcf466f" is_pre_election: true
I20260926 03:48:50.873817 19378 leader_election.cc:302] T e1b536ee9f0e4c6da21bc1c9781e400e P ee200939409f42beb9d62636bb226e77 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 916b0426be81457ca54b37874c6b4612, ee200939409f42beb9d62636bb226e77; no voters:
I20260926 03:48:50.874095 19310 raft_consensus.cc:2454] T e1b536ee9f0e4c6da21bc1c9781e400e P bb322be6e99448b3afae9e92cfcf466f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee200939409f42beb9d62636bb226e77 in term 0.
I20260926 03:48:50.874256 19514 raft_consensus.cc:2781] T e1b536ee9f0e4c6da21bc1c9781e400e P ee200939409f42beb9d62636bb226e77 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 03:48:50.874454 19514 raft_consensus.cc:482] T e1b536ee9f0e4c6da21bc1c9781e400e P ee200939409f42beb9d62636bb226e77 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 03:48:50.874756 19514 raft_consensus.cc:3037] T e1b536ee9f0e4c6da21bc1c9781e400e P ee200939409f42beb9d62636bb226e77 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:48:50.877939 19514 raft_consensus.cc:504] T e1b536ee9f0e4c6da21bc1c9781e400e P ee200939409f42beb9d62636bb226e77 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb322be6e99448b3afae9e92cfcf466f" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 33525 } } peers { permanent_uuid: "ee200939409f42beb9d62636bb226e77" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 35147 } } peers { permanent_uuid: "916b0426be81457ca54b37874c6b4612" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 40699 } }
I20260926 03:48:50.878569 19514 leader_election.cc:288] T e1b536ee9f0e4c6da21bc1c9781e400e P ee200939409f42beb9d62636bb226e77 [CANDIDATE]: Term 1 election: Requested vote from peers bb322be6e99448b3afae9e92cfcf466f (127.0.194.194:33525), 916b0426be81457ca54b37874c6b4612 (127.0.194.193:40699)
I20260926 03:48:50.879091 19310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1b536ee9f0e4c6da21bc1c9781e400e" candidate_uuid: "ee200939409f42beb9d62636bb226e77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb322be6e99448b3afae9e92cfcf466f"
I20260926 03:48:50.879082 19176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1b536ee9f0e4c6da21bc1c9781e400e" candidate_uuid: "ee200939409f42beb9d62636bb226e77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "916b0426be81457ca54b37874c6b4612"
I20260926 03:48:50.879344 19310 raft_consensus.cc:3037] T e1b536ee9f0e4c6da21bc1c9781e400e P bb322be6e99448b3afae9e92cfcf466f [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:48:50.879374 19176 raft_consensus.cc:3037] T e1b536ee9f0e4c6da21bc1c9781e400e P 916b0426be81457ca54b37874c6b4612 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:48:50.881656 19176 raft_consensus.cc:2454] T e1b536ee9f0e4c6da21bc1c9781e400e P 916b0426be81457ca54b37874c6b4612 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee200939409f42beb9d62636bb226e77 in term 1.
I20260926 03:48:50.881769 19310 raft_consensus.cc:2454] T e1b536ee9f0e4c6da21bc1c9781e400e P bb322be6e99448b3afae9e92cfcf466f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee200939409f42beb9d62636bb226e77 in term 1.
I20260926 03:48:50.882136 19378 leader_election.cc:302] T e1b536ee9f0e4c6da21bc1c9781e400e P ee200939409f42beb9d62636bb226e77 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 916b0426be81457ca54b37874c6b4612, ee200939409f42beb9d62636bb226e77; no voters:
I20260926 03:48:50.882418 19514 raft_consensus.cc:2781] T e1b536ee9f0e4c6da21bc1c9781e400e P ee200939409f42beb9d62636bb226e77 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:48:50.882889 19514 raft_consensus.cc:686] T e1b536ee9f0e4c6da21bc1c9781e400e P ee200939409f42beb9d62636bb226e77 [term 1 LEADER]: Becoming Leader. State: Replica: ee200939409f42beb9d62636bb226e77, State: Running, Role: LEADER
I20260926 03:48:50.883754 19514 consensus_queue.cc:237] T e1b536ee9f0e4c6da21bc1c9781e400e P ee200939409f42beb9d62636bb226e77 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb322be6e99448b3afae9e92cfcf466f" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 33525 } } peers { permanent_uuid: "ee200939409f42beb9d62636bb226e77" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 35147 } } peers { permanent_uuid: "916b0426be81457ca54b37874c6b4612" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 40699 } }
I20260926 03:48:50.888286 19032 catalog_manager.cc:5495] T e1b536ee9f0e4c6da21bc1c9781e400e P ee200939409f42beb9d62636bb226e77 reported cstate change: term changed from 0 to 1, leader changed from <none> to ee200939409f42beb9d62636bb226e77 (127.0.194.195). New cstate: current_term: 1 leader_uuid: "ee200939409f42beb9d62636bb226e77" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb322be6e99448b3afae9e92cfcf466f" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 33525 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee200939409f42beb9d62636bb226e77" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 35147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "916b0426be81457ca54b37874c6b4612" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 40699 } health_report { overall_health: UNKNOWN } } }
I20260926 03:48:50.918973 19504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:50.913744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53116 (local address 127.0.194.195:35147)
0926 03:48:50.913941 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0926 03:48:50.913945 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:48:50.914012 (+ 67us) server_negotiation.cc:407] Connection header received
0926 03:48:50.914189 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:48:50.914193 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:48:50.914270 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:48:50.914404 (+ 134us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:48:50.914824 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:48:50.915546 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:48:50.916923 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:48:50.917303 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:48:50.917394 (+ 91us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:48:50.917930 (+ 536us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:48:50.918422 (+ 492us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:48:50.918551 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0926 03:48:50.918659 (+ 108us) server_negotiation.cc:299] Negotiation successful
0926 03:48:50.918702 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 03:48:50.928154 19310 raft_consensus.cc:1261] T e1b536ee9f0e4c6da21bc1c9781e400e P bb322be6e99448b3afae9e92cfcf466f [term 1 FOLLOWER]: Refusing update from remote peer ee200939409f42beb9d62636bb226e77: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:48:50.928154 19176 raft_consensus.cc:1261] T e1b536ee9f0e4c6da21bc1c9781e400e P 916b0426be81457ca54b37874c6b4612 [term 1 FOLLOWER]: Refusing update from remote peer ee200939409f42beb9d62636bb226e77: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:48:50.929237 19517 consensus_queue.cc:1035] T e1b536ee9f0e4c6da21bc1c9781e400e P ee200939409f42beb9d62636bb226e77 [LEADER]: Connected to new peer: Peer: permanent_uuid: "916b0426be81457ca54b37874c6b4612" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 40699 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:48:50.929625 19514 consensus_queue.cc:1035] T e1b536ee9f0e4c6da21bc1c9781e400e P ee200939409f42beb9d62636bb226e77 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb322be6e99448b3afae9e92cfcf466f" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 33525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:48:50.949230 19522 mvcc.cc:205] Tried to move back new op lower bound from 7333455998670671872 to 7333455998503956480. Current Snapshot: MvccSnapshot[applied={T|T < 7333455998670671872}]
I20260926 03:48:50.960702 19031 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34400:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:48:50.961035 19031 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34400:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:48:50.965170 19031 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 46d29631621547398f8f7e5451d9f243: Sending DeleteTablet for 3 replicas of tablet e1b536ee9f0e4c6da21bc1c9781e400e
I20260926 03:48:50.966490 19424 tablet_service.cc:1514] Processing DeleteTablet for tablet e1b536ee9f0e4c6da21bc1c9781e400e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:48:50 UTC) from {username='kudu', principal='kudu/127.0.194.254@KRBTEST.COM'} at 127.0.0.1:53102
I20260926 03:48:50.966481 19290 tablet_service.cc:1514] Processing DeleteTablet for tablet e1b536ee9f0e4c6da21bc1c9781e400e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:48:50 UTC) from {username='kudu', principal='kudu/127.0.194.254@KRBTEST.COM'} at 127.0.0.1:40026
I20260926 03:48:50.967782 19156 tablet_service.cc:1514] Processing DeleteTablet for tablet e1b536ee9f0e4c6da21bc1c9781e400e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:48:50 UTC) from {username='kudu', principal='kudu/127.0.194.254@KRBTEST.COM'} at 127.0.0.1:55056
I20260926 03:48:50.970480 779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu pid 19093
I20260926 03:48:50.977686 19532 tablet_replica.cc:330] T e1b536ee9f0e4c6da21bc1c9781e400e P 916b0426be81457ca54b37874c6b4612: stopping tablet replica
I20260926 03:48:50.978901 19532 raft_consensus.cc:2229] T e1b536ee9f0e4c6da21bc1c9781e400e P 916b0426be81457ca54b37874c6b4612 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:48:50.979977 19530 tablet_replica.cc:330] T e1b536ee9f0e4c6da21bc1c9781e400e P bb322be6e99448b3afae9e92cfcf466f: stopping tablet replica
I20260926 03:48:50.980829 19532 raft_consensus.cc:2258] T e1b536ee9f0e4c6da21bc1c9781e400e P 916b0426be81457ca54b37874c6b4612 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:48:50.981510 19530 raft_consensus.cc:2229] T e1b536ee9f0e4c6da21bc1c9781e400e P bb322be6e99448b3afae9e92cfcf466f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:48:50.982590 19530 raft_consensus.cc:2258] T e1b536ee9f0e4c6da21bc1c9781e400e P bb322be6e99448b3afae9e92cfcf466f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:48:50.985321 19532 ts_tablet_manager.cc:1905] T e1b536ee9f0e4c6da21bc1c9781e400e P 916b0426be81457ca54b37874c6b4612: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:48:50.985599 19531 tablet_replica.cc:330] T e1b536ee9f0e4c6da21bc1c9781e400e P ee200939409f42beb9d62636bb226e77: stopping tablet replica
I20260926 03:48:50.986043 19531 raft_consensus.cc:2229] T e1b536ee9f0e4c6da21bc1c9781e400e P ee200939409f42beb9d62636bb226e77 [term 1 LEADER]: Raft consensus shutting down.
I20260926 03:48:50.986903 19531 raft_consensus.cc:2258] T e1b536ee9f0e4c6da21bc1c9781e400e P ee200939409f42beb9d62636bb226e77 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:48:50.988497 19530 ts_tablet_manager.cc:1905] T e1b536ee9f0e4c6da21bc1c9781e400e P bb322be6e99448b3afae9e92cfcf466f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:48:50.988915 19531 ts_tablet_manager.cc:1905] T e1b536ee9f0e4c6da21bc1c9781e400e P ee200939409f42beb9d62636bb226e77: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 26 03:48:50 dist-test-slave-n2th krb5kdc[18987](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394528, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.194.193@KRBTEST.COM
I20260926 03:48:50.996369 19531 ts_tablet_manager.cc:1918] T e1b536ee9f0e4c6da21bc1c9781e400e P ee200939409f42beb9d62636bb226e77: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:48:50.996585 19531 log.cc:1196] T e1b536ee9f0e4c6da21bc1c9781e400e P ee200939409f42beb9d62636bb226e77: Deleting WAL directory at /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790394221564982-779-0/minicluster-data/ts-2/wal/wals/e1b536ee9f0e4c6da21bc1c9781e400e
I20260926 03:48:50.997159 19531 ts_tablet_manager.cc:1939] T e1b536ee9f0e4c6da21bc1c9781e400e P ee200939409f42beb9d62636bb226e77: Deleting consensus metadata
I20260926 03:48:50.999164 19017 catalog_manager.cc:4841] TS ee200939409f42beb9d62636bb226e77 (127.0.194.195:35147): tablet e1b536ee9f0e4c6da21bc1c9781e400e (table test-table [id=85f920ff16ff43968bd07dbe020d622c]) successfully deleted
I20260926 03:48:51.001384 19532 ts_tablet_manager.cc:1918] T e1b536ee9f0e4c6da21bc1c9781e400e P 916b0426be81457ca54b37874c6b4612: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:48:51.001652 19532 log.cc:1196] T e1b536ee9f0e4c6da21bc1c9781e400e P 916b0426be81457ca54b37874c6b4612: Deleting WAL directory at /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790394221564982-779-0/minicluster-data/ts-0/wal/wals/e1b536ee9f0e4c6da21bc1c9781e400e
I20260926 03:48:51.002188 19530 ts_tablet_manager.cc:1918] T e1b536ee9f0e4c6da21bc1c9781e400e P bb322be6e99448b3afae9e92cfcf466f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:48:51.002388 19530 log.cc:1196] T e1b536ee9f0e4c6da21bc1c9781e400e P bb322be6e99448b3afae9e92cfcf466f: Deleting WAL directory at /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790394221564982-779-0/minicluster-data/ts-1/wal/wals/e1b536ee9f0e4c6da21bc1c9781e400e
I20260926 03:48:51.002497 19532 ts_tablet_manager.cc:1939] T e1b536ee9f0e4c6da21bc1c9781e400e P 916b0426be81457ca54b37874c6b4612: Deleting consensus metadata
I20260926 03:48:51.003007 19530 ts_tablet_manager.cc:1939] T e1b536ee9f0e4c6da21bc1c9781e400e P bb322be6e99448b3afae9e92cfcf466f: Deleting consensus metadata
I20260926 03:48:51.004693 19017 catalog_manager.cc:4841] TS 916b0426be81457ca54b37874c6b4612 (127.0.194.193:40699): tablet e1b536ee9f0e4c6da21bc1c9781e400e (table test-table [id=85f920ff16ff43968bd07dbe020d622c]) successfully deleted
I20260926 03:48:51.004743 19019 catalog_manager.cc:4841] TS bb322be6e99448b3afae9e92cfcf466f (127.0.194.194:33525): tablet e1b536ee9f0e4c6da21bc1c9781e400e (table test-table [id=85f920ff16ff43968bd07dbe020d622c]) successfully deleted
I20260926 03:48:51.005108 19506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:50.971965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55066 (local address 127.0.194.193:40699)
0926 03:48:50.972202 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0926 03:48:50.972212 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 03:48:50.982839 (+ 10627us) server_negotiation.cc:407] Connection header received
0926 03:48:50.983382 (+ 543us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:48:50.983389 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:48:50.983503 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:48:50.983590 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:48:50.985575 (+ 1985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:48:50.986777 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:48:50.989082 (+ 2305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:48:50.989746 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:48:50.989855 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:48:50.996119 (+ 6264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:48:50.996146 (+ 27us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:48:50.996169 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:48:50.996226 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:48:51.000676 (+ 4450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:48:51.001747 (+ 1071us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:48:51.001757 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:48:51.001761 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:48:51.001952 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:48:51.002492 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:48:51.002499 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:48:51.002502 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:48:51.003036 (+ 534us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:48:51.003351 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0926 03:48:51.004058 (+ 707us) server_negotiation.cc:299] Negotiation successful
0926 03:48:51.004317 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260926 03:48:51.006323 19216 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:48:51.098907 779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu with pid 19093
I20260926 03:48:51.123325 779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu pid 19227
Sep 26 03:48:51 dist-test-slave-n2th krb5kdc[18987](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394528, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.194.194@KRBTEST.COM
I20260926 03:48:51.139770 19502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:51.124153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40042 (local address 127.0.194.194:33525)
0926 03:48:51.124339 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0926 03:48:51.124347 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:48:51.124404 (+ 57us) server_negotiation.cc:407] Connection header received
0926 03:48:51.124607 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:48:51.124613 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:48:51.124700 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:48:51.124813 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:48:51.126263 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:48:51.127243 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:48:51.128592 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:48:51.129024 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:48:51.129160 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:48:51.133922 (+ 4762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:48:51.133938 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:48:51.133956 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:48:51.133994 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:48:51.136847 (+ 2853us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:48:51.137571 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:48:51.137578 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:48:51.137581 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:48:51.137717 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:48:51.138103 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:48:51.138108 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:48:51.138111 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:48:51.138493 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:48:51.138819 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0926 03:48:51.139166 (+ 347us) server_negotiation.cc:299] Negotiation successful
0926 03:48:51.139413 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260926 03:48:51.140399 19350 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:48:51.229878 779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu with pid 19227
I20260926 03:48:51.247270 779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu pid 19361
Sep 26 03:48:51 dist-test-slave-n2th krb5kdc[18987](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394528, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.194.195@KRBTEST.COM
I20260926 03:48:51.265017 19504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:51.248049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53118 (local address 127.0.194.195:35147)
0926 03:48:51.248254 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0926 03:48:51.248260 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:48:51.248354 (+ 94us) server_negotiation.cc:407] Connection header received
0926 03:48:51.248563 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:48:51.248567 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:48:51.248635 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:48:51.248738 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:48:51.250022 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:48:51.250909 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:48:51.252171 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:48:51.252639 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:48:51.252791 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:48:51.257584 (+ 4793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:48:51.257600 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:48:51.257619 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:48:51.257669 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:48:51.262015 (+ 4346us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:48:51.262771 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:48:51.262782 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:48:51.262785 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:48:51.262976 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:48:51.263342 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:48:51.263346 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:48:51.263351 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:48:51.263819 (+ 468us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:48:51.264156 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0926 03:48:51.264349 (+ 193us) server_negotiation.cc:299] Negotiation successful
0926 03:48:51.264598 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106}
I20260926 03:48:51.265810 19484 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:48:51.387152 779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu with pid 19361
I20260926 03:48:51.406116 779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu pid 19000
I20260926 03:48:51.407298 19062 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:48:51.516709 779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu with pid 19000
2026-09-26T03:48:51Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4687 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-26T03:48:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:48:51Z Disabled control of system clock
I20260926 03:48:51.595500 779 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
/tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790394221564982-779-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790394221564982-779-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790394221564982-779-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790394221564982-779-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.194.254:39197
--webserver_interface=127.0.194.254
--webserver_port=0
--builtin_ntp_servers=127.0.194.212:40621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.194.254:39197
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 03:48:51.942129 19541 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:48:51.942441 19541 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:48:51.942600 19541 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:48:51.950413 19541 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:48:51.950552 19541 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:48:51.950655 19541 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:48:51.950779 19541 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:48:51.950846 19541 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:48:51.960897 19541 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.194.212:40621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790394221564982-779-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790394221564982-779-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.194.254:39197
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.194.254:39197
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790394221564982-779-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.194.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790394221564982-779-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:48:51.962249 19541 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:48:51.964226 19541 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:48:51.976689 19546 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:48:51.976742 19547 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:48:51.976712 19549 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:48:51.977535 19541 server_base.cc:1034] running on GCE node
I20260926 03:48:51.978382 19541 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:48:51.979974 19541 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:48:51.981179 19541 hybrid_clock.cc:648] HybridClock initialized: now 1790394531981136 us; error 36 us; skew 500 ppm
I20260926 03:48:51.981501 19541 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 03:48:51.989930 19541 webserver.cc:460] Webserver started at http://127.0.194.254:38377/ using document root <none> and password file <none>
I20260926 03:48:51.990672 19541 fs_manager.cc:359] Metadata directory not provided
I20260926 03:48:51.990841 19541 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:48:51.991205 19541 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:48:51.994288 19541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790394221564982-779-0/minicluster-data/master-0/data/instance:
uuid: "af358612e91248dea4d6064642b3bbc8"
format_stamp: "Formatted at 2026-09-26 03:48:51 on dist-test-slave-n2th"
server_key: "51ba5e3a1688686bf74cc9c0cf8c0e58"
server_key_iv: "2ea17c255fe3b3574588dbfd756b7a88"
server_key_version: "encryptionkey@0"
I20260926 03:48:51.995196 19541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790394221564982-779-0/minicluster-data/master-0/wal/instance:
uuid: "af358612e91248dea4d6064642b3bbc8"
format_stamp: "Formatted at 2026-09-26 03:48:51 on dist-test-slave-n2th"
server_key: "51ba5e3a1688686bf74cc9c0cf8c0e58"
server_key_iv: "2ea17c255fe3b3574588dbfd756b7a88"
server_key_version: "encryptionkey@0"
I20260926 03:48:52.000792 19541 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 03:48:52.004587 19555 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:48:52.006238 19541 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 03:48:52.006475 19541 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790394221564982-779-0/minicluster-data/master-0/data,/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790394221564982-779-0/minicluster-data/master-0/wal
uuid: "af358612e91248dea4d6064642b3bbc8"
format_stamp: "Formatted at 2026-09-26 03:48:51 on dist-test-slave-n2th"
server_key: "51ba5e3a1688686bf74cc9c0cf8c0e58"
server_key_iv: "2ea17c255fe3b3574588dbfd756b7a88"
server_key_version: "encryptionkey@0"
I20260926 03:48:52.006774 19541 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790394221564982-779-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790394221564982-779-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790394221564982-779-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:48:52.022018 19541 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:48:52.022979 19541 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:48:52.023283 19541 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:48:52.042430 19541 rpc_server.cc:307] RPC server started. Bound to: 127.0.194.254:39197
I20260926 03:48:52.042451 19606 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.194.254:39197 every 8 connection(s)
I20260926 03:48:52.043926 19541 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790394221564982-779-0/minicluster-data/master-0/data/info.pb
I20260926 03:48:52.047901 19607 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:48:52.052775 779 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu as pid 19541
I20260926 03:48:52.052965 779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790394221564982-779-0/minicluster-data/master-0/wal/instance
I20260926 03:48:52.053556 779 external_mini_cluster.cc:1428] Setting key 7b9074103ca24241dd66e3eae5a62472
I20260926 03:48:52.060091 19607 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af358612e91248dea4d6064642b3bbc8: Bootstrap starting.
I20260926 03:48:52.063491 19607 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P af358612e91248dea4d6064642b3bbc8: Neither blocks nor log segments found. Creating new log.
I20260926 03:48:52.064612 19607 log.cc:824] T 00000000000000000000000000000000 P af358612e91248dea4d6064642b3bbc8: Log is configured to *not* fsync() on all Append() calls
I20260926 03:48:52.068296 19610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:52.055741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33668 (local address 127.0.194.254:39197)
0926 03:48:52.056810 (+ 1069us) server_negotiation.cc:206] Beginning negotiation
0926 03:48:52.056824 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 03:48:52.056872 (+ 48us) server_negotiation.cc:407] Connection header received
0926 03:48:52.058829 (+ 1957us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:48:52.058856 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:48:52.059401 (+ 545us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:48:52.059803 (+ 402us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:48:52.060391 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:48:52.061731 (+ 1340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:48:52.063159 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:48:52.063963 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:48:52.064160 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:48:52.065462 (+ 1302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:48:52.065478 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:48:52.065495 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:48:52.066138 (+ 643us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:48:52.066246 (+ 108us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:48:52.066273 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:48:52.066396 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0926 03:48:52.066616 (+ 220us) server_negotiation.cc:299] Negotiation successful
0926 03:48:52.067014 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":620,"thread_start_us":369,"threads_started":1}
I20260926 03:48:52.068291 19607 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af358612e91248dea4d6064642b3bbc8: No bootstrap required, opened a new log
I20260926 03:48:52.076756 19607 raft_consensus.cc:348] T 00000000000000000000000000000000 P af358612e91248dea4d6064642b3bbc8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af358612e91248dea4d6064642b3bbc8" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 39197 } }
I20260926 03:48:52.077152 19607 raft_consensus.cc:374] T 00000000000000000000000000000000 P af358612e91248dea4d6064642b3bbc8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:48:52.077229 19607 raft_consensus.cc:729] T 00000000000000000000000000000000 P af358612e91248dea4d6064642b3bbc8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af358612e91248dea4d6064642b3bbc8, State: Initialized, Role: FOLLOWER
I20260926 03:48:52.077811 19607 consensus_queue.cc:260] T 00000000000000000000000000000000 P af358612e91248dea4d6064642b3bbc8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af358612e91248dea4d6064642b3bbc8" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 39197 } }
I20260926 03:48:52.078037 19607 raft_consensus.cc:388] T 00000000000000000000000000000000 P af358612e91248dea4d6064642b3bbc8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:48:52.078164 19607 raft_consensus.cc:482] T 00000000000000000000000000000000 P af358612e91248dea4d6064642b3bbc8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:48:52.078361 19607 raft_consensus.cc:3037] T 00000000000000000000000000000000 P af358612e91248dea4d6064642b3bbc8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:48:52.080822 19607 raft_consensus.cc:504] T 00000000000000000000000000000000 P af358612e91248dea4d6064642b3bbc8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af358612e91248dea4d6064642b3bbc8" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 39197 } }
I20260926 03:48:52.081295 19607 leader_election.cc:302] T 00000000000000000000000000000000 P af358612e91248dea4d6064642b3bbc8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: af358612e91248dea4d6064642b3bbc8; no voters:
I20260926 03:48:52.081866 19607 leader_election.cc:288] T 00000000000000000000000000000000 P af358612e91248dea4d6064642b3bbc8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:48:52.082031 19612 raft_consensus.cc:2781] T 00000000000000000000000000000000 P af358612e91248dea4d6064642b3bbc8 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:48:52.082753 19612 raft_consensus.cc:686] T 00000000000000000000000000000000 P af358612e91248dea4d6064642b3bbc8 [term 1 LEADER]: Becoming Leader. State: Replica: af358612e91248dea4d6064642b3bbc8, State: Running, Role: LEADER
I20260926 03:48:52.083240 19612 consensus_queue.cc:237] T 00000000000000000000000000000000 P af358612e91248dea4d6064642b3bbc8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af358612e91248dea4d6064642b3bbc8" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 39197 } }
I20260926 03:48:52.084022 19607 sys_catalog.cc:564] T 00000000000000000000000000000000 P af358612e91248dea4d6064642b3bbc8 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:48:52.086443 19613 sys_catalog.cc:455] T 00000000000000000000000000000000 P af358612e91248dea4d6064642b3bbc8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "af358612e91248dea4d6064642b3bbc8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af358612e91248dea4d6064642b3bbc8" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 39197 } } }
I20260926 03:48:52.086344 19614 sys_catalog.cc:455] T 00000000000000000000000000000000 P af358612e91248dea4d6064642b3bbc8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader af358612e91248dea4d6064642b3bbc8. Latest consensus state: current_term: 1 leader_uuid: "af358612e91248dea4d6064642b3bbc8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af358612e91248dea4d6064642b3bbc8" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 39197 } } }
I20260926 03:48:52.086824 19614 sys_catalog.cc:458] T 00000000000000000000000000000000 P af358612e91248dea4d6064642b3bbc8 [sys.catalog]: This master's current role is: LEADER
I20260926 03:48:52.088631 19613 sys_catalog.cc:458] T 00000000000000000000000000000000 P af358612e91248dea4d6064642b3bbc8 [sys.catalog]: This master's current role is: LEADER
I20260926 03:48:52.089176 19619 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:48:52.094486 19619 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:48:52.103955 19619 catalog_manager.cc:1347] Generated new cluster ID: aa5afa56ead34a37873a32c490c56576
I20260926 03:48:52.104096 19619 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:48:52.118508 19619 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:48:52.120627 19619 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:48:52.132584 19619 catalog_manager.cc:5868] T 00000000000000000000000000000000 P af358612e91248dea4d6064642b3bbc8: Generated new TSK 0
I20260926 03:48:52.133515 19619 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 03:48:52.143998 779 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
/tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790394221564982-779-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790394221564982-779-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790394221564982-779-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790394221564982-779-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.194.193:0
--local_ip_for_outbound_sockets=127.0.194.193
--webserver_interface=127.0.194.193
--webserver_port=0
--tserver_master_addrs=127.0.194.254:39197
--builtin_ntp_servers=127.0.194.212:40621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 03:48:52.514333 19631 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:48:52.514642 19631 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:48:52.514829 19631 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:48:52.522392 19631 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:48:52.522501 19631 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:48:52.522668 19631 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.194.193
I20260926 03:48:52.532065 19631 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.194.212:40621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790394221564982-779-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790394221564982-779-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.194.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790394221564982-779-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.194.193
--webserver_port=0
--tserver_master_addrs=127.0.194.254:39197
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.194.193
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790394221564982-779-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:48:52.533376 19631 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:48:52.535229 19631 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:48:52.548600 19637 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:48:52.548681 19636 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:48:52.548678 19639 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:48:52.549423 19631 server_base.cc:1034] running on GCE node
I20260926 03:48:52.550136 19631 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:48:52.551221 19631 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:48:52.552451 19631 hybrid_clock.cc:648] HybridClock initialized: now 1790394532552394 us; error 61 us; skew 500 ppm
I20260926 03:48:52.552788 19631 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 03:48:52.555320 19631 webserver.cc:460] Webserver started at http://127.0.194.193:43757/ using document root <none> and password file <none>
I20260926 03:48:52.556068 19631 fs_manager.cc:359] Metadata directory not provided
I20260926 03:48:52.556200 19631 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:48:52.556540 19631 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:48:52.559604 19631 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790394221564982-779-0/minicluster-data/ts-0/data/instance:
uuid: "9e400a3e7b074d67bfb7f882ea213a21"
format_stamp: "Formatted at 2026-09-26 03:48:52 on dist-test-slave-n2th"
server_key: "991c6f95276fb012f0921b6d75a992bc"
server_key_iv: "bec991f3ae2e15c1cdd48bc848e52b3c"
server_key_version: "encryptionkey@0"
I20260926 03:48:52.560417 19631 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790394221564982-779-0/minicluster-data/ts-0/wal/instance:
uuid: "9e400a3e7b074d67bfb7f882ea213a21"
format_stamp: "Formatted at 2026-09-26 03:48:52 on dist-test-slave-n2th"
server_key: "991c6f95276fb012f0921b6d75a992bc"
server_key_iv: "bec991f3ae2e15c1cdd48bc848e52b3c"
server_key_version: "encryptionkey@0"
I20260926 03:48:52.565784 19631 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 03:48:52.569607 19645 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:48:52.571110 19631 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 03:48:52.571290 19631 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790394221564982-779-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790394221564982-779-0/minicluster-data/ts-0/wal
uuid: "9e400a3e7b074d67bfb7f882ea213a21"
format_stamp: "Formatted at 2026-09-26 03:48:52 on dist-test-slave-n2th"
server_key: "991c6f95276fb012f0921b6d75a992bc"
server_key_iv: "bec991f3ae2e15c1cdd48bc848e52b3c"
server_key_version: "encryptionkey@0"
I20260926 03:48:52.571537 19631 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790394221564982-779-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790394221564982-779-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790394221564982-779-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:48:52.591236 19631 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:48:52.592197 19631 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:48:52.592522 19631 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:48:52.593587 19631 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:48:52.595129 19631 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:48:52.595254 19631 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:48:52.595371 19631 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:48:52.595463 19631 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:48:52.628767 19631 rpc_server.cc:307] RPC server started. Bound to: 127.0.194.193:38591
I20260926 03:48:52.628825 19757 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.194.193:38591 every 8 connection(s)
I20260926 03:48:52.630194 19631 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790394221564982-779-0/minicluster-data/ts-0/data/info.pb
I20260926 03:48:52.634279 779 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu as pid 19631
I20260926 03:48:52.634475 779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790394221564982-779-0/minicluster-data/ts-0/wal/instance
I20260926 03:48:52.635067 779 external_mini_cluster.cc:1428] Setting key b33645bf0d459a38dab831475f83b896
I20260926 03:48:52.638239 779 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
/tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790394221564982-779-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790394221564982-779-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790394221564982-779-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790394221564982-779-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.194.194:0
--local_ip_for_outbound_sockets=127.0.194.194
--webserver_interface=127.0.194.194
--webserver_port=0
--tserver_master_addrs=127.0.194.254:39197
--builtin_ntp_servers=127.0.194.212:40621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 03:48:52.645215 19760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:52.632948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.193:38471 (local address 127.0.194.254:39197)
0926 03:48:52.633387 (+ 439us) server_negotiation.cc:206] Beginning negotiation
0926 03:48:52.633394 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:48:52.636742 (+ 3348us) server_negotiation.cc:407] Connection header received
0926 03:48:52.637292 (+ 550us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:48:52.637296 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:48:52.637359 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:48:52.637448 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:48:52.638973 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:48:52.639672 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:48:52.642169 (+ 2497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:48:52.642582 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:48:52.642684 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:48:52.644436 (+ 1752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:48:52.644440 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:48:52.644443 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:48:52.644466 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:48:52.644505 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:48:52.644517 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:48:52.644584 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 03:48:52.644851 (+ 267us) server_negotiation.cc:299] Negotiation successful
0926 03:48:52.644935 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":194,"threads_started":1}
I20260926 03:48:52.646808 19761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:52.633619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:39197 (local address 127.0.194.193:38471)
0926 03:48:52.634786 (+ 1167us) negotiation.cc:107] Waiting for socket to connect
0926 03:48:52.634836 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 03:48:52.636980 (+ 2144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:48:52.637741 (+ 761us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:48:52.637753 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:48:52.637887 (+ 134us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:48:52.638790 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:48:52.638821 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:48:52.639824 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:48:52.639833 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:48:52.642000 (+ 2167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:48:52.642018 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:48:52.642849 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:48:52.642858 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:48:52.643216 (+ 358us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:48:52.644030 (+ 814us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 03:48:52.644035 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:48:52.644093 (+ 58us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 03:48:52.644217 (+ 124us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:48:52.644610 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:48:52.644624 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:48:52.644635 (+ 11us) client_negotiation.cc:769] Sending connection context
0926 03:48:52.644826 (+ 191us) client_negotiation.cc:240] Negotiation successful
0926 03:48:52.645289 (+ 463us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":537,"thread_start_us":323,"threads_started":1}
I20260926 03:48:52.648633 19758 heartbeater.cc:347] Connected to a master server at 127.0.194.254:39197
I20260926 03:48:52.648962 19758 heartbeater.cc:464] Registering TS with master...
I20260926 03:48:52.649756 19758 heartbeater.cc:511] Master 127.0.194.254:39197 requested a full tablet report, sending...
I20260926 03:48:52.652271 19572 ts_manager.cc:194] Registered new tserver with Master: 9e400a3e7b074d67bfb7f882ea213a21 (127.0.194.193:38591)
I20260926 03:48:52.654512 19572 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.194.193:38471
W20260926 03:48:52.987545 19762 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:48:52.987869 19762 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:48:52.987962 19762 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:48:52.995837 19762 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:48:52.995959 19762 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:48:52.996069 19762 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.194.194
I20260926 03:48:53.005686 19762 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.194.212:40621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790394221564982-779-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790394221564982-779-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.194.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790394221564982-779-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.194.194
--webserver_port=0
--tserver_master_addrs=127.0.194.254:39197
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.194.194
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790394221564982-779-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:48:53.007081 19762 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:48:53.008857 19762 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:48:53.022662 19768 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:48:53.022693 19767 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:48:53.022969 19770 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:48:53.023638 19762 server_base.cc:1034] running on GCE node
I20260926 03:48:53.024294 19762 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:48:53.025307 19762 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:48:53.026542 19762 hybrid_clock.cc:648] HybridClock initialized: now 1790394533026477 us; error 61 us; skew 500 ppm
I20260926 03:48:53.026932 19762 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 03:48:53.029408 19762 webserver.cc:460] Webserver started at http://127.0.194.194:44285/ using document root <none> and password file <none>
I20260926 03:48:53.030145 19762 fs_manager.cc:359] Metadata directory not provided
I20260926 03:48:53.030277 19762 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:48:53.030606 19762 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:48:53.033618 19762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790394221564982-779-0/minicluster-data/ts-1/data/instance:
uuid: "c405c65ff3fe4685a238d380d3a6e71e"
format_stamp: "Formatted at 2026-09-26 03:48:53 on dist-test-slave-n2th"
server_key: "dff30d8c0e25add3a7396ef7446ca6a5"
server_key_iv: "45c3b51cfd08a2133f72fec374cc0441"
server_key_version: "encryptionkey@0"
I20260926 03:48:53.034432 19762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790394221564982-779-0/minicluster-data/ts-1/wal/instance:
uuid: "c405c65ff3fe4685a238d380d3a6e71e"
format_stamp: "Formatted at 2026-09-26 03:48:53 on dist-test-slave-n2th"
server_key: "dff30d8c0e25add3a7396ef7446ca6a5"
server_key_iv: "45c3b51cfd08a2133f72fec374cc0441"
server_key_version: "encryptionkey@0"
I20260926 03:48:53.039825 19762 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 03:48:53.043846 19776 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:48:53.045508 19762 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 03:48:53.045686 19762 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790394221564982-779-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790394221564982-779-0/minicluster-data/ts-1/wal
uuid: "c405c65ff3fe4685a238d380d3a6e71e"
format_stamp: "Formatted at 2026-09-26 03:48:53 on dist-test-slave-n2th"
server_key: "dff30d8c0e25add3a7396ef7446ca6a5"
server_key_iv: "45c3b51cfd08a2133f72fec374cc0441"
server_key_version: "encryptionkey@0"
I20260926 03:48:53.045918 19762 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790394221564982-779-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790394221564982-779-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790394221564982-779-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:48:53.072551 19762 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:48:53.073390 19762 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:48:53.073676 19762 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:48:53.074628 19762 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:48:53.076349 19762 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:48:53.076478 19762 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:48:53.076573 19762 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:48:53.076632 19762 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:48:53.111025 19762 rpc_server.cc:307] RPC server started. Bound to: 127.0.194.194:34891
I20260926 03:48:53.111114 19888 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.194.194:34891 every 8 connection(s)
I20260926 03:48:53.112367 19762 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790394221564982-779-0/minicluster-data/ts-1/data/info.pb
I20260926 03:48:53.118420 779 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu as pid 19762
I20260926 03:48:53.118628 779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790394221564982-779-0/minicluster-data/ts-1/wal/instance
I20260926 03:48:53.119203 779 external_mini_cluster.cc:1428] Setting key f5d927a6240f87f98d1344dd6e468c8f
I20260926 03:48:53.122704 779 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
/tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790394221564982-779-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790394221564982-779-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790394221564982-779-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790394221564982-779-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.194.195:0
--local_ip_for_outbound_sockets=127.0.194.195
--webserver_interface=127.0.194.195
--webserver_port=0
--tserver_master_addrs=127.0.194.254:39197
--builtin_ntp_servers=127.0.194.212:40621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 03:48:53.127341 19760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:53.115009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.194:37751 (local address 127.0.194.254:39197)
0926 03:48:53.115196 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0926 03:48:53.115200 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:48:53.116581 (+ 1381us) server_negotiation.cc:407] Connection header received
0926 03:48:53.118694 (+ 2113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:48:53.118702 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:48:53.118836 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:48:53.118927 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:48:53.120433 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:48:53.121529 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:48:53.124205 (+ 2676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:48:53.124640 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:48:53.124745 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:48:53.126551 (+ 1806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:48:53.126556 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:48:53.126560 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:48:53.126580 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:48:53.126612 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:48:53.126624 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:48:53.126696 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 03:48:53.127017 (+ 321us) server_negotiation.cc:299] Negotiation successful
0926 03:48:53.127106 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 03:48:53.129148 19891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:53.115491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:39197 (local address 127.0.194.194:37751)
0926 03:48:53.116435 (+ 944us) negotiation.cc:107] Waiting for socket to connect
0926 03:48:53.116474 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 03:48:53.118302 (+ 1828us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:48:53.119238 (+ 936us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:48:53.119253 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:48:53.119402 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:48:53.120235 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:48:53.120259 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:48:53.121703 (+ 1444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:48:53.121709 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:48:53.124020 (+ 2311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:48:53.124043 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:48:53.124897 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:48:53.124906 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:48:53.125299 (+ 393us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:48:53.126129 (+ 830us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 03:48:53.126135 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:48:53.126197 (+ 62us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 03:48:53.126329 (+ 132us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:48:53.126765 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:48:53.126780 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:48:53.126791 (+ 11us) client_negotiation.cc:769] Sending connection context
0926 03:48:53.127003 (+ 212us) client_negotiation.cc:240] Negotiation successful
0926 03:48:53.127500 (+ 497us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":281,"threads_started":1}
I20260926 03:48:53.130710 19889 heartbeater.cc:347] Connected to a master server at 127.0.194.254:39197
I20260926 03:48:53.131115 19889 heartbeater.cc:464] Registering TS with master...
I20260926 03:48:53.132047 19889 heartbeater.cc:511] Master 127.0.194.254:39197 requested a full tablet report, sending...
I20260926 03:48:53.133543 19572 ts_manager.cc:194] Registered new tserver with Master: c405c65ff3fe4685a238d380d3a6e71e (127.0.194.194:34891)
I20260926 03:48:53.134393 19572 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.194.194:37751
W20260926 03:48:53.460192 19892 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:48:53.460556 19892 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:48:53.460716 19892 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:48:53.468122 19892 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:48:53.468272 19892 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:48:53.468441 19892 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.194.195
I20260926 03:48:53.477909 19892 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.194.212:40621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790394221564982-779-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790394221564982-779-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.194.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790394221564982-779-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.194.195
--webserver_port=0
--tserver_master_addrs=127.0.194.254:39197
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.194.195
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790394221564982-779-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:48:53.479367 19892 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:48:53.481146 19892 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:48:53.494429 19900 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:48:53.494517 19898 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:48:53.494494 19897 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:48:53.495374 19892 server_base.cc:1034] running on GCE node
I20260926 03:48:53.495992 19892 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:48:53.496974 19892 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:48:53.498208 19892 hybrid_clock.cc:648] HybridClock initialized: now 1790394533498150 us; error 50 us; skew 500 ppm
I20260926 03:48:53.498554 19892 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 03:48:53.500820 19892 webserver.cc:460] Webserver started at http://127.0.194.195:41985/ using document root <none> and password file <none>
I20260926 03:48:53.501540 19892 fs_manager.cc:359] Metadata directory not provided
I20260926 03:48:53.501669 19892 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:48:53.502002 19892 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:48:53.505043 19892 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790394221564982-779-0/minicluster-data/ts-2/data/instance:
uuid: "ffebd0cde82845a38c83bbf2c9bf29e1"
format_stamp: "Formatted at 2026-09-26 03:48:53 on dist-test-slave-n2th"
server_key: "590407a24640a857f6651b8dc8b7dd46"
server_key_iv: "f1d22e8069e50b8a8e10c24ea4bccf91"
server_key_version: "encryptionkey@0"
I20260926 03:48:53.505859 19892 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790394221564982-779-0/minicluster-data/ts-2/wal/instance:
uuid: "ffebd0cde82845a38c83bbf2c9bf29e1"
format_stamp: "Formatted at 2026-09-26 03:48:53 on dist-test-slave-n2th"
server_key: "590407a24640a857f6651b8dc8b7dd46"
server_key_iv: "f1d22e8069e50b8a8e10c24ea4bccf91"
server_key_version: "encryptionkey@0"
I20260926 03:48:53.511708 19892 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20260926 03:48:53.515468 19906 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:48:53.517031 19892 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 03:48:53.517212 19892 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790394221564982-779-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790394221564982-779-0/minicluster-data/ts-2/wal
uuid: "ffebd0cde82845a38c83bbf2c9bf29e1"
format_stamp: "Formatted at 2026-09-26 03:48:53 on dist-test-slave-n2th"
server_key: "590407a24640a857f6651b8dc8b7dd46"
server_key_iv: "f1d22e8069e50b8a8e10c24ea4bccf91"
server_key_version: "encryptionkey@0"
I20260926 03:48:53.517458 19892 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790394221564982-779-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790394221564982-779-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790394221564982-779-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:48:53.534096 19892 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:48:53.535046 19892 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:48:53.535357 19892 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:48:53.536334 19892 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:48:53.537791 19892 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:48:53.537899 19892 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:48:53.538036 19892 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:48:53.538125 19892 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:48:53.574015 19892 rpc_server.cc:307] RPC server started. Bound to: 127.0.194.195:38801
I20260926 03:48:53.574038 20018 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.194.195:38801 every 8 connection(s)
I20260926 03:48:53.575489 19892 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790394221564982-779-0/minicluster-data/ts-2/data/info.pb
I20260926 03:48:53.580946 779 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu as pid 19892
I20260926 03:48:53.581138 779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790394221564982-779-0/minicluster-data/ts-2/wal/instance
I20260926 03:48:53.581591 779 external_mini_cluster.cc:1428] Setting key 732e2d886c6a827ddc4f31a7e29df76c
I20260926 03:48:53.588763 19760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:53.578030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.195:37291 (local address 127.0.194.254:39197)
0926 03:48:53.578204 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0926 03:48:53.578208 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:48:53.579768 (+ 1560us) server_negotiation.cc:407] Connection header received
0926 03:48:53.581686 (+ 1918us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:48:53.581693 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:48:53.581786 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:48:53.581904 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:48:53.583275 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:48:53.584158 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:48:53.585860 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:48:53.586271 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:48:53.586403 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:48:53.587902 (+ 1499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:48:53.587907 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:48:53.587910 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:48:53.587931 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:48:53.587964 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:48:53.587976 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:48:53.588070 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0926 03:48:53.588302 (+ 232us) server_negotiation.cc:299] Negotiation successful
0926 03:48:53.588405 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 03:48:53.589795 20021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:53.578581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:39197 (local address 127.0.194.195:37291)
0926 03:48:53.579610 (+ 1029us) negotiation.cc:107] Waiting for socket to connect
0926 03:48:53.579645 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 03:48:53.581396 (+ 1751us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:48:53.582229 (+ 833us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:48:53.582238 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:48:53.582338 (+ 100us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:48:53.583102 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:48:53.583122 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:48:53.584310 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:48:53.584315 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:48:53.585700 (+ 1385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:48:53.585713 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:48:53.586509 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:48:53.586517 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:48:53.586826 (+ 309us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:48:53.587560 (+ 734us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 03:48:53.587562 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:48:53.587605 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 03:48:53.587692 (+ 87us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:48:53.588068 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:48:53.588081 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:48:53.588092 (+ 11us) client_negotiation.cc:769] Sending connection context
0926 03:48:53.588274 (+ 182us) client_negotiation.cc:240] Negotiation successful
0926 03:48:53.588711 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":565,"thread_start_us":384,"threads_started":1}
I20260926 03:48:53.591248 20019 heartbeater.cc:347] Connected to a master server at 127.0.194.254:39197
I20260926 03:48:53.591586 20019 heartbeater.cc:464] Registering TS with master...
I20260926 03:48:53.592406 20019 heartbeater.cc:511] Master 127.0.194.254:39197 requested a full tablet report, sending...
I20260926 03:48:53.593864 19572 ts_manager.cc:194] Registered new tserver with Master: ffebd0cde82845a38c83bbf2c9bf29e1 (127.0.194.195:38801)
I20260926 03:48:53.594614 19572 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.194.195:37291
I20260926 03:48:53.596982 779 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 03:48:53.606904 19760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:53.601197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33676 (local address 127.0.194.254:39197)
0926 03:48:53.601385 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0926 03:48:53.601389 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:48:53.601763 (+ 374us) server_negotiation.cc:407] Connection header received
0926 03:48:53.601906 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:48:53.601910 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:48:53.601992 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:48:53.602077 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:48:53.602514 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:48:53.603280 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:48:53.604605 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:48:53.604999 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:48:53.605098 (+ 99us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:48:53.606198 (+ 1100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:48:53.606203 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:48:53.606206 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:48:53.606224 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:48:53.606256 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:48:53.606267 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:48:53.606382 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0926 03:48:53.606557 (+ 175us) server_negotiation.cc:299] Negotiation successful
0926 03:48:53.606606 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 03:48:53.611028 19572 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33676:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 03:48:53.618067 19572 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 03:48:53.642911 20033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:53.630795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.194:34891 (local address 127.0.0.1:44128)
0926 03:48:53.632528 (+ 1733us) negotiation.cc:107] Waiting for socket to connect
0926 03:48:53.632585 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0926 03:48:53.632924 (+ 339us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:48:53.634170 (+ 1246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:48:53.634181 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:48:53.634235 (+ 54us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:48:53.634800 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:48:53.634817 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:48:53.637234 (+ 2417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:48:53.637239 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:48:53.639625 (+ 2386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:48:53.639642 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:48:53.641925 (+ 2283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:48:53.641933 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:48:53.642159 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:48:53.642219 (+ 60us) client_negotiation.cc:769] Sending connection context
0926 03:48:53.642347 (+ 128us) client_negotiation.cc:240] Negotiation successful
0926 03:48:53.642542 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1431,"thread_start_us":6083,"threads_started":1}
I20260926 03:48:53.643733 20032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:53.631185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44128 (local address 127.0.194.194:34891)
0926 03:48:53.631741 (+ 556us) server_negotiation.cc:206] Beginning negotiation
0926 03:48:53.631754 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 03:48:53.632718 (+ 964us) server_negotiation.cc:407] Connection header received
0926 03:48:53.633211 (+ 493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:48:53.633224 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:48:53.633543 (+ 319us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:48:53.633738 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:48:53.635056 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:48:53.637096 (+ 2040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:48:53.639931 (+ 2835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:48:53.641400 (+ 1469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:48:53.641523 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:48:53.641787 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0926 03:48:53.643192 (+ 1405us) server_negotiation.cc:299] Negotiation successful
0926 03:48:53.643384 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":272,"threads_started":1}
I20260926 03:48:53.647742 20031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:53.631294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.193:38591 (local address 127.0.0.1:44742)
0926 03:48:53.637117 (+ 5823us) negotiation.cc:107] Waiting for socket to connect
0926 03:48:53.637132 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 03:48:53.637277 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:48:53.638184 (+ 907us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:48:53.638194 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:48:53.638221 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:48:53.639872 (+ 1651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:48:53.639888 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:48:53.641758 (+ 1870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:48:53.641763 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:48:53.644969 (+ 3206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:48:53.644981 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:48:53.646979 (+ 1998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:48:53.646985 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:48:53.647160 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:48:53.647204 (+ 44us) client_negotiation.cc:769] Sending connection context
0926 03:48:53.647291 (+ 87us) client_negotiation.cc:240] Negotiation successful
0926 03:48:53.647435 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5717,"thread_start_us":765,"threads_started":1}
I20260926 03:48:53.648005 20034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:53.631831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44742 (local address 127.0.194.193:38591)
0926 03:48:53.636809 (+ 4978us) server_negotiation.cc:206] Beginning negotiation
0926 03:48:53.636820 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 03:48:53.637275 (+ 455us) server_negotiation.cc:407] Connection header received
0926 03:48:53.637502 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:48:53.637512 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:48:53.637756 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:48:53.637943 (+ 187us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:48:53.640075 (+ 2132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:48:53.641603 (+ 1528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:48:53.645173 (+ 3570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:48:53.646718 (+ 1545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:48:53.646934 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:48:53.647221 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0926 03:48:53.647359 (+ 138us) server_negotiation.cc:299] Negotiation successful
0926 03:48:53.647513 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4800,"thread_start_us":4705,"threads_started":1}
I20260926 03:48:53.649595 19824 tablet_service.cc:1467] Processing CreateTablet for tablet 290c4d09d2354d388b3eef0215a5a83e (DEFAULT_TABLE table=test-table [id=52a01d4eacf44f0e8c40580df8535521]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:48:53.651173 19824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 290c4d09d2354d388b3eef0215a5a83e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:48:53.651788 19693 tablet_service.cc:1467] Processing CreateTablet for tablet 290c4d09d2354d388b3eef0215a5a83e (DEFAULT_TABLE table=test-table [id=52a01d4eacf44f0e8c40580df8535521]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:48:53.653290 19693 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 290c4d09d2354d388b3eef0215a5a83e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:48:53.653630 20033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:53.637519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.195:38801 (local address 127.0.0.1:50838)
0926 03:48:53.644089 (+ 6570us) negotiation.cc:107] Waiting for socket to connect
0926 03:48:53.644109 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 03:48:53.644213 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:48:53.646649 (+ 2436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:48:53.646654 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:48:53.646683 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:48:53.646882 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:48:53.646893 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:48:53.649057 (+ 2164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:48:53.649062 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:48:53.650875 (+ 1813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:48:53.650891 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:48:53.652741 (+ 1850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:48:53.652750 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:48:53.652942 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:48:53.652997 (+ 55us) client_negotiation.cc:769] Sending connection context
0926 03:48:53.653095 (+ 98us) client_negotiation.cc:240] Negotiation successful
0926 03:48:53.653242 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6439,"thread_start_us":7760,"threads_started":1}
I20260926 03:48:53.653682 20036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:53.637541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50838 (local address 127.0.194.195:38801)
0926 03:48:53.645926 (+ 8385us) server_negotiation.cc:206] Beginning negotiation
0926 03:48:53.645937 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 03:48:53.645972 (+ 35us) server_negotiation.cc:407] Connection header received
0926 03:48:53.646074 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:48:53.646082 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:48:53.646342 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:48:53.646566 (+ 224us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:48:53.647081 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:48:53.648871 (+ 1790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:48:53.651047 (+ 2176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:48:53.652512 (+ 1465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:48:53.652662 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:48:53.652948 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0926 03:48:53.653188 (+ 240us) server_negotiation.cc:299] Negotiation successful
0926 03:48:53.653366 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8213,"thread_start_us":8071,"threads_started":1}
I20260926 03:48:53.657835 19758 heartbeater.cc:503] Master 127.0.194.254:39197 was elected leader, sending a full tablet report...
I20260926 03:48:53.662021 19954 tablet_service.cc:1467] Processing CreateTablet for tablet 290c4d09d2354d388b3eef0215a5a83e (DEFAULT_TABLE table=test-table [id=52a01d4eacf44f0e8c40580df8535521]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:48:53.663658 19954 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 290c4d09d2354d388b3eef0215a5a83e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:48:53.664641 20037 tablet_bootstrap.cc:492] T 290c4d09d2354d388b3eef0215a5a83e P c405c65ff3fe4685a238d380d3a6e71e: Bootstrap starting.
I20260926 03:48:53.666654 20038 tablet_bootstrap.cc:492] T 290c4d09d2354d388b3eef0215a5a83e P 9e400a3e7b074d67bfb7f882ea213a21: Bootstrap starting.
I20260926 03:48:53.668431 20037 tablet_bootstrap.cc:654] T 290c4d09d2354d388b3eef0215a5a83e P c405c65ff3fe4685a238d380d3a6e71e: Neither blocks nor log segments found. Creating new log.
I20260926 03:48:53.669457 20037 log.cc:824] T 290c4d09d2354d388b3eef0215a5a83e P c405c65ff3fe4685a238d380d3a6e71e: Log is configured to *not* fsync() on all Append() calls
I20260926 03:48:53.670203 20038 tablet_bootstrap.cc:654] T 290c4d09d2354d388b3eef0215a5a83e P 9e400a3e7b074d67bfb7f882ea213a21: Neither blocks nor log segments found. Creating new log.
I20260926 03:48:53.671490 20038 log.cc:824] T 290c4d09d2354d388b3eef0215a5a83e P 9e400a3e7b074d67bfb7f882ea213a21: Log is configured to *not* fsync() on all Append() calls
I20260926 03:48:53.673594 20037 tablet_bootstrap.cc:492] T 290c4d09d2354d388b3eef0215a5a83e P c405c65ff3fe4685a238d380d3a6e71e: No bootstrap required, opened a new log
I20260926 03:48:53.673885 20037 ts_tablet_manager.cc:1397] T 290c4d09d2354d388b3eef0215a5a83e P c405c65ff3fe4685a238d380d3a6e71e: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260926 03:48:53.674791 20038 tablet_bootstrap.cc:492] T 290c4d09d2354d388b3eef0215a5a83e P 9e400a3e7b074d67bfb7f882ea213a21: No bootstrap required, opened a new log
I20260926 03:48:53.675065 20038 ts_tablet_manager.cc:1397] T 290c4d09d2354d388b3eef0215a5a83e P 9e400a3e7b074d67bfb7f882ea213a21: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.005s
I20260926 03:48:53.679188 20041 tablet_bootstrap.cc:492] T 290c4d09d2354d388b3eef0215a5a83e P ffebd0cde82845a38c83bbf2c9bf29e1: Bootstrap starting.
I20260926 03:48:53.683311 20041 tablet_bootstrap.cc:654] T 290c4d09d2354d388b3eef0215a5a83e P ffebd0cde82845a38c83bbf2c9bf29e1: Neither blocks nor log segments found. Creating new log.
I20260926 03:48:53.683036 20037 raft_consensus.cc:348] T 290c4d09d2354d388b3eef0215a5a83e P c405c65ff3fe4685a238d380d3a6e71e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c405c65ff3fe4685a238d380d3a6e71e" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 34891 } } peers { permanent_uuid: "ffebd0cde82845a38c83bbf2c9bf29e1" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 38801 } } peers { permanent_uuid: "9e400a3e7b074d67bfb7f882ea213a21" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 38591 } }
I20260926 03:48:53.683568 20037 raft_consensus.cc:374] T 290c4d09d2354d388b3eef0215a5a83e P c405c65ff3fe4685a238d380d3a6e71e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:48:53.683679 20037 raft_consensus.cc:729] T 290c4d09d2354d388b3eef0215a5a83e P c405c65ff3fe4685a238d380d3a6e71e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c405c65ff3fe4685a238d380d3a6e71e, State: Initialized, Role: FOLLOWER
I20260926 03:48:53.684496 20037 consensus_queue.cc:260] T 290c4d09d2354d388b3eef0215a5a83e P c405c65ff3fe4685a238d380d3a6e71e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c405c65ff3fe4685a238d380d3a6e71e" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 34891 } } peers { permanent_uuid: "ffebd0cde82845a38c83bbf2c9bf29e1" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 38801 } } peers { permanent_uuid: "9e400a3e7b074d67bfb7f882ea213a21" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 38591 } }
I20260926 03:48:53.684609 20038 raft_consensus.cc:348] T 290c4d09d2354d388b3eef0215a5a83e P 9e400a3e7b074d67bfb7f882ea213a21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c405c65ff3fe4685a238d380d3a6e71e" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 34891 } } peers { permanent_uuid: "ffebd0cde82845a38c83bbf2c9bf29e1" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 38801 } } peers { permanent_uuid: "9e400a3e7b074d67bfb7f882ea213a21" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 38591 } }
I20260926 03:48:53.685045 20038 raft_consensus.cc:374] T 290c4d09d2354d388b3eef0215a5a83e P 9e400a3e7b074d67bfb7f882ea213a21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:48:53.685002 20041 log.cc:824] T 290c4d09d2354d388b3eef0215a5a83e P ffebd0cde82845a38c83bbf2c9bf29e1: Log is configured to *not* fsync() on all Append() calls
I20260926 03:48:53.685158 20038 raft_consensus.cc:729] T 290c4d09d2354d388b3eef0215a5a83e P 9e400a3e7b074d67bfb7f882ea213a21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e400a3e7b074d67bfb7f882ea213a21, State: Initialized, Role: FOLLOWER
I20260926 03:48:53.685956 19889 heartbeater.cc:503] Master 127.0.194.254:39197 was elected leader, sending a full tablet report...
I20260926 03:48:53.685887 20038 consensus_queue.cc:260] T 290c4d09d2354d388b3eef0215a5a83e P 9e400a3e7b074d67bfb7f882ea213a21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c405c65ff3fe4685a238d380d3a6e71e" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 34891 } } peers { permanent_uuid: "ffebd0cde82845a38c83bbf2c9bf29e1" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 38801 } } peers { permanent_uuid: "9e400a3e7b074d67bfb7f882ea213a21" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 38591 } }
I20260926 03:48:53.686973 20037 ts_tablet_manager.cc:1428] T 290c4d09d2354d388b3eef0215a5a83e P c405c65ff3fe4685a238d380d3a6e71e: Time spent starting tablet: real 0.013s user 0.010s sys 0.005s
I20260926 03:48:53.688975 20038 ts_tablet_manager.cc:1428] T 290c4d09d2354d388b3eef0215a5a83e P 9e400a3e7b074d67bfb7f882ea213a21: Time spent starting tablet: real 0.014s user 0.001s sys 0.012s
I20260926 03:48:53.692904 20041 tablet_bootstrap.cc:492] T 290c4d09d2354d388b3eef0215a5a83e P ffebd0cde82845a38c83bbf2c9bf29e1: No bootstrap required, opened a new log
I20260926 03:48:53.693243 20041 ts_tablet_manager.cc:1397] T 290c4d09d2354d388b3eef0215a5a83e P ffebd0cde82845a38c83bbf2c9bf29e1: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.004s
I20260926 03:48:53.701256 20041 raft_consensus.cc:348] T 290c4d09d2354d388b3eef0215a5a83e P ffebd0cde82845a38c83bbf2c9bf29e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c405c65ff3fe4685a238d380d3a6e71e" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 34891 } } peers { permanent_uuid: "ffebd0cde82845a38c83bbf2c9bf29e1" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 38801 } } peers { permanent_uuid: "9e400a3e7b074d67bfb7f882ea213a21" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 38591 } }
I20260926 03:48:53.701650 20041 raft_consensus.cc:374] T 290c4d09d2354d388b3eef0215a5a83e P ffebd0cde82845a38c83bbf2c9bf29e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:48:53.701731 20041 raft_consensus.cc:729] T 290c4d09d2354d388b3eef0215a5a83e P ffebd0cde82845a38c83bbf2c9bf29e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ffebd0cde82845a38c83bbf2c9bf29e1, State: Initialized, Role: FOLLOWER
I20260926 03:48:53.702265 20041 consensus_queue.cc:260] T 290c4d09d2354d388b3eef0215a5a83e P ffebd0cde82845a38c83bbf2c9bf29e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c405c65ff3fe4685a238d380d3a6e71e" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 34891 } } peers { permanent_uuid: "ffebd0cde82845a38c83bbf2c9bf29e1" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 38801 } } peers { permanent_uuid: "9e400a3e7b074d67bfb7f882ea213a21" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 38591 } }
I20260926 03:48:53.703323 20019 heartbeater.cc:503] Master 127.0.194.254:39197 was elected leader, sending a full tablet report...
I20260926 03:48:53.703811 20041 ts_tablet_manager.cc:1428] T 290c4d09d2354d388b3eef0215a5a83e P ffebd0cde82845a38c83bbf2c9bf29e1: Time spent starting tablet: real 0.010s user 0.012s sys 0.000s
I20260926 03:48:53.714506 20042 raft_consensus.cc:482] T 290c4d09d2354d388b3eef0215a5a83e P c405c65ff3fe4685a238d380d3a6e71e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 03:48:53.714915 20042 raft_consensus.cc:504] T 290c4d09d2354d388b3eef0215a5a83e P c405c65ff3fe4685a238d380d3a6e71e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c405c65ff3fe4685a238d380d3a6e71e" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 34891 } } peers { permanent_uuid: "ffebd0cde82845a38c83bbf2c9bf29e1" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 38801 } } peers { permanent_uuid: "9e400a3e7b074d67bfb7f882ea213a21" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 38591 } }
I20260926 03:48:53.717116 20042 leader_election.cc:288] T 290c4d09d2354d388b3eef0215a5a83e P c405c65ff3fe4685a238d380d3a6e71e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ffebd0cde82845a38c83bbf2c9bf29e1 (127.0.194.195:38801), 9e400a3e7b074d67bfb7f882ea213a21 (127.0.194.193:38591)
I20260926 03:48:53.728143 20034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:53.717892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.194:50721 (local address 127.0.194.193:38591)
0926 03:48:53.718610 (+ 718us) server_negotiation.cc:206] Beginning negotiation
0926 03:48:53.718622 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 03:48:53.720268 (+ 1646us) server_negotiation.cc:407] Connection header received
0926 03:48:53.720555 (+ 287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:48:53.720561 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:48:53.720649 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:48:53.720710 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:48:53.721613 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:48:53.722623 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:48:53.725743 (+ 3120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:48:53.727034 (+ 1291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:48:53.727206 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:48:53.727276 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 03:48:53.727749 (+ 473us) server_negotiation.cc:299] Negotiation successful
0926 03:48:53.727883 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319}
I20260926 03:48:53.728143 20036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:53.717917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.194:39759 (local address 127.0.194.195:38801)
0926 03:48:53.718122 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0926 03:48:53.718129 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:48:53.719979 (+ 1850us) server_negotiation.cc:407] Connection header received
0926 03:48:53.720210 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:48:53.720218 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:48:53.720309 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:48:53.720405 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:48:53.721282 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:48:53.722524 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:48:53.725743 (+ 3219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:48:53.727038 (+ 1295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:48:53.727206 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:48:53.727276 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 03:48:53.727749 (+ 473us) server_negotiation.cc:299] Negotiation successful
0926 03:48:53.727882 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 03:48:53.728302 20047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:53.718276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.193:38591 (local address 127.0.194.194:50721)
0926 03:48:53.720146 (+ 1870us) negotiation.cc:107] Waiting for socket to connect
0926 03:48:53.720168 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 03:48:53.720384 (+ 216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:48:53.720919 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:48:53.720926 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:48:53.720960 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:48:53.721389 (+ 429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:48:53.721402 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:48:53.722765 (+ 1363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:48:53.722771 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:48:53.725503 (+ 2732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:48:53.725519 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:48:53.727254 (+ 1735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:48:53.727260 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:48:53.727506 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:48:53.727592 (+ 86us) client_negotiation.cc:769] Sending connection context
0926 03:48:53.727683 (+ 91us) client_negotiation.cc:240] Negotiation successful
0926 03:48:53.727908 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1718,"mutex_wait_us":296,"thread_start_us":1057,"threads_started":1}
I20260926 03:48:53.728322 20046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:53.717473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.195:38801 (local address 127.0.194.194:39759)
0926 03:48:53.719850 (+ 2377us) negotiation.cc:107] Waiting for socket to connect
0926 03:48:53.719870 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 03:48:53.720057 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:48:53.720540 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:48:53.720546 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:48:53.720572 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:48:53.721097 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:48:53.721110 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:48:53.722652 (+ 1542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:48:53.722656 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:48:53.725504 (+ 2848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:48:53.725519 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:48:53.727253 (+ 1734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:48:53.727260 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:48:53.727512 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:48:53.727589 (+ 77us) client_negotiation.cc:769] Sending connection context
0926 03:48:53.727683 (+ 94us) client_negotiation.cc:240] Negotiation successful
0926 03:48:53.727908 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2201,"mutex_wait_us":277,"thread_start_us":1451,"threads_started":1}
I20260926 03:48:53.729434 19974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "290c4d09d2354d388b3eef0215a5a83e" candidate_uuid: "c405c65ff3fe4685a238d380d3a6e71e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ffebd0cde82845a38c83bbf2c9bf29e1" is_pre_election: true
I20260926 03:48:53.729460 19713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "290c4d09d2354d388b3eef0215a5a83e" candidate_uuid: "c405c65ff3fe4685a238d380d3a6e71e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e400a3e7b074d67bfb7f882ea213a21" is_pre_election: true
I20260926 03:48:53.729951 19974 raft_consensus.cc:2454] T 290c4d09d2354d388b3eef0215a5a83e P ffebd0cde82845a38c83bbf2c9bf29e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c405c65ff3fe4685a238d380d3a6e71e in term 0.
I20260926 03:48:53.729952 19713 raft_consensus.cc:2454] T 290c4d09d2354d388b3eef0215a5a83e P 9e400a3e7b074d67bfb7f882ea213a21 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c405c65ff3fe4685a238d380d3a6e71e in term 0.
I20260926 03:48:53.730603 19778 leader_election.cc:302] T 290c4d09d2354d388b3eef0215a5a83e P c405c65ff3fe4685a238d380d3a6e71e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c405c65ff3fe4685a238d380d3a6e71e, ffebd0cde82845a38c83bbf2c9bf29e1; no voters:
I20260926 03:48:53.731038 20042 raft_consensus.cc:2781] T 290c4d09d2354d388b3eef0215a5a83e P c405c65ff3fe4685a238d380d3a6e71e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 03:48:53.731196 20042 raft_consensus.cc:482] T 290c4d09d2354d388b3eef0215a5a83e P c405c65ff3fe4685a238d380d3a6e71e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 03:48:53.731313 20042 raft_consensus.cc:3037] T 290c4d09d2354d388b3eef0215a5a83e P c405c65ff3fe4685a238d380d3a6e71e [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:48:53.733506 20042 raft_consensus.cc:504] T 290c4d09d2354d388b3eef0215a5a83e P c405c65ff3fe4685a238d380d3a6e71e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c405c65ff3fe4685a238d380d3a6e71e" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 34891 } } peers { permanent_uuid: "ffebd0cde82845a38c83bbf2c9bf29e1" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 38801 } } peers { permanent_uuid: "9e400a3e7b074d67bfb7f882ea213a21" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 38591 } }
I20260926 03:48:53.734076 20042 leader_election.cc:288] T 290c4d09d2354d388b3eef0215a5a83e P c405c65ff3fe4685a238d380d3a6e71e [CANDIDATE]: Term 1 election: Requested vote from peers ffebd0cde82845a38c83bbf2c9bf29e1 (127.0.194.195:38801), 9e400a3e7b074d67bfb7f882ea213a21 (127.0.194.193:38591)
I20260926 03:48:53.734612 19974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "290c4d09d2354d388b3eef0215a5a83e" candidate_uuid: "c405c65ff3fe4685a238d380d3a6e71e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ffebd0cde82845a38c83bbf2c9bf29e1"
I20260926 03:48:53.734612 19713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "290c4d09d2354d388b3eef0215a5a83e" candidate_uuid: "c405c65ff3fe4685a238d380d3a6e71e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e400a3e7b074d67bfb7f882ea213a21"
I20260926 03:48:53.734850 19713 raft_consensus.cc:3037] T 290c4d09d2354d388b3eef0215a5a83e P 9e400a3e7b074d67bfb7f882ea213a21 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:48:53.734863 19974 raft_consensus.cc:3037] T 290c4d09d2354d388b3eef0215a5a83e P ffebd0cde82845a38c83bbf2c9bf29e1 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:48:53.738165 19974 raft_consensus.cc:2454] T 290c4d09d2354d388b3eef0215a5a83e P ffebd0cde82845a38c83bbf2c9bf29e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c405c65ff3fe4685a238d380d3a6e71e in term 1.
I20260926 03:48:53.738165 19713 raft_consensus.cc:2454] T 290c4d09d2354d388b3eef0215a5a83e P 9e400a3e7b074d67bfb7f882ea213a21 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c405c65ff3fe4685a238d380d3a6e71e in term 1.
I20260926 03:48:53.738674 19780 leader_election.cc:302] T 290c4d09d2354d388b3eef0215a5a83e P c405c65ff3fe4685a238d380d3a6e71e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9e400a3e7b074d67bfb7f882ea213a21, c405c65ff3fe4685a238d380d3a6e71e; no voters:
I20260926 03:48:53.739151 20042 raft_consensus.cc:2781] T 290c4d09d2354d388b3eef0215a5a83e P c405c65ff3fe4685a238d380d3a6e71e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:48:53.739598 20042 raft_consensus.cc:686] T 290c4d09d2354d388b3eef0215a5a83e P c405c65ff3fe4685a238d380d3a6e71e [term 1 LEADER]: Becoming Leader. State: Replica: c405c65ff3fe4685a238d380d3a6e71e, State: Running, Role: LEADER
I20260926 03:48:53.740108 20042 consensus_queue.cc:237] T 290c4d09d2354d388b3eef0215a5a83e P c405c65ff3fe4685a238d380d3a6e71e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c405c65ff3fe4685a238d380d3a6e71e" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 34891 } } peers { permanent_uuid: "ffebd0cde82845a38c83bbf2c9bf29e1" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 38801 } } peers { permanent_uuid: "9e400a3e7b074d67bfb7f882ea213a21" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 38591 } }
I20260926 03:48:53.744886 19572 catalog_manager.cc:5495] T 290c4d09d2354d388b3eef0215a5a83e P c405c65ff3fe4685a238d380d3a6e71e reported cstate change: term changed from 0 to 1, leader changed from <none> to c405c65ff3fe4685a238d380d3a6e71e (127.0.194.194). New cstate: current_term: 1 leader_uuid: "c405c65ff3fe4685a238d380d3a6e71e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c405c65ff3fe4685a238d380d3a6e71e" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 34891 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ffebd0cde82845a38c83bbf2c9bf29e1" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 38801 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9e400a3e7b074d67bfb7f882ea213a21" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 38591 } health_report { overall_health: UNKNOWN } } }
I20260926 03:48:53.768356 20032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:53.762980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44134 (local address 127.0.194.194:34891)
0926 03:48:53.763158 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0926 03:48:53.763163 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:48:53.763181 (+ 18us) server_negotiation.cc:407] Connection header received
0926 03:48:53.763397 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:48:53.763402 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:48:53.763500 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:48:53.763643 (+ 143us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:48:53.764060 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:48:53.764784 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:48:53.766258 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:48:53.766658 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:48:53.766833 (+ 175us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:48:53.767286 (+ 453us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:48:53.767772 (+ 486us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:48:53.767904 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0926 03:48:53.768110 (+ 206us) server_negotiation.cc:299] Negotiation successful
0926 03:48:53.768148 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 03:48:53.776958 19713 raft_consensus.cc:1261] T 290c4d09d2354d388b3eef0215a5a83e P 9e400a3e7b074d67bfb7f882ea213a21 [term 1 FOLLOWER]: Refusing update from remote peer c405c65ff3fe4685a238d380d3a6e71e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:48:53.776958 19974 raft_consensus.cc:1261] T 290c4d09d2354d388b3eef0215a5a83e P ffebd0cde82845a38c83bbf2c9bf29e1 [term 1 FOLLOWER]: Refusing update from remote peer c405c65ff3fe4685a238d380d3a6e71e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:48:53.778074 20042 consensus_queue.cc:1035] T 290c4d09d2354d388b3eef0215a5a83e P c405c65ff3fe4685a238d380d3a6e71e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e400a3e7b074d67bfb7f882ea213a21" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 38591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:48:53.778399 20048 consensus_queue.cc:1035] T 290c4d09d2354d388b3eef0215a5a83e P c405c65ff3fe4685a238d380d3a6e71e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ffebd0cde82845a38c83bbf2c9bf29e1" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 38801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:48:53.816378 19572 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33676:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:48:53.816645 19572 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33676:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:48:53.820081 19572 catalog_manager.cc:5782] T 00000000000000000000000000000000 P af358612e91248dea4d6064642b3bbc8: Sending DeleteTablet for 3 replicas of tablet 290c4d09d2354d388b3eef0215a5a83e
I20260926 03:48:53.821326 19824 tablet_service.cc:1514] Processing DeleteTablet for tablet 290c4d09d2354d388b3eef0215a5a83e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:48:53 UTC) from {username='slave'} at 127.0.0.1:44128
I20260926 03:48:53.821365 19954 tablet_service.cc:1514] Processing DeleteTablet for tablet 290c4d09d2354d388b3eef0215a5a83e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:48:53 UTC) from {username='slave'} at 127.0.0.1:50838
I20260926 03:48:53.821548 19693 tablet_service.cc:1514] Processing DeleteTablet for tablet 290c4d09d2354d388b3eef0215a5a83e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:48:53 UTC) from {username='slave'} at 127.0.0.1:44742
I20260926 03:48:53.822489 20062 tablet_replica.cc:330] T 290c4d09d2354d388b3eef0215a5a83e P 9e400a3e7b074d67bfb7f882ea213a21: stopping tablet replica
I20260926 03:48:53.823071 779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu pid 19631
I20260926 03:48:53.823027 20062 raft_consensus.cc:2229] T 290c4d09d2354d388b3eef0215a5a83e P 9e400a3e7b074d67bfb7f882ea213a21 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:48:53.823674 20062 raft_consensus.cc:2258] T 290c4d09d2354d388b3eef0215a5a83e P 9e400a3e7b074d67bfb7f882ea213a21 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:48:53.825942 20062 ts_tablet_manager.cc:1905] T 290c4d09d2354d388b3eef0215a5a83e P 9e400a3e7b074d67bfb7f882ea213a21: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:48:53.834669 20062 ts_tablet_manager.cc:1918] T 290c4d09d2354d388b3eef0215a5a83e P 9e400a3e7b074d67bfb7f882ea213a21: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:48:53.835009 20062 log.cc:1196] T 290c4d09d2354d388b3eef0215a5a83e P 9e400a3e7b074d67bfb7f882ea213a21: Deleting WAL directory at /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790394221564982-779-0/minicluster-data/ts-0/wal/wals/290c4d09d2354d388b3eef0215a5a83e
I20260926 03:48:53.835671 20062 ts_tablet_manager.cc:1939] T 290c4d09d2354d388b3eef0215a5a83e P 9e400a3e7b074d67bfb7f882ea213a21: Deleting consensus metadata
I20260926 03:48:53.836714 20061 tablet_replica.cc:330] T 290c4d09d2354d388b3eef0215a5a83e P ffebd0cde82845a38c83bbf2c9bf29e1: stopping tablet replica
I20260926 03:48:53.837294 20061 raft_consensus.cc:2229] T 290c4d09d2354d388b3eef0215a5a83e P ffebd0cde82845a38c83bbf2c9bf29e1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:48:53.837906 20061 raft_consensus.cc:2258] T 290c4d09d2354d388b3eef0215a5a83e P ffebd0cde82845a38c83bbf2c9bf29e1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:48:53.838250 19559 catalog_manager.cc:4841] TS 9e400a3e7b074d67bfb7f882ea213a21 (127.0.194.193:38591): tablet 290c4d09d2354d388b3eef0215a5a83e (table test-table [id=52a01d4eacf44f0e8c40580df8535521]) successfully deleted
I20260926 03:48:53.838793 20060 tablet_replica.cc:330] T 290c4d09d2354d388b3eef0215a5a83e P c405c65ff3fe4685a238d380d3a6e71e: stopping tablet replica
I20260926 03:48:53.839587 20060 raft_consensus.cc:2229] T 290c4d09d2354d388b3eef0215a5a83e P c405c65ff3fe4685a238d380d3a6e71e [term 1 LEADER]: Raft consensus shutting down.
I20260926 03:48:53.840394 20060 raft_consensus.cc:2258] T 290c4d09d2354d388b3eef0215a5a83e P c405c65ff3fe4685a238d380d3a6e71e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:48:53.841964 20061 ts_tablet_manager.cc:1905] T 290c4d09d2354d388b3eef0215a5a83e P ffebd0cde82845a38c83bbf2c9bf29e1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:48:53.844403 20060 ts_tablet_manager.cc:1905] T 290c4d09d2354d388b3eef0215a5a83e P c405c65ff3fe4685a238d380d3a6e71e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:48:53.847779 20034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:53.825755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44746 (local address 127.0.194.193:38591)
0926 03:48:53.825936 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0926 03:48:53.825944 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:48:53.839337 (+ 13393us) server_negotiation.cc:407] Connection header received
0926 03:48:53.839575 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:48:53.839583 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:48:53.839696 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:48:53.839806 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:48:53.840496 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:48:53.841635 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:48:53.843623 (+ 1988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:48:53.844247 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:48:53.844375 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:48:53.845916 (+ 1541us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:48:53.845927 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:48:53.845943 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:48:53.846844 (+ 901us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:48:53.846912 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:48:53.846937 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:48:53.847024 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0926 03:48:53.847273 (+ 249us) server_negotiation.cc:299] Negotiation successful
0926 03:48:53.847410 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 03:48:53.849030 19753 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:48:53.850014 20061 ts_tablet_manager.cc:1918] T 290c4d09d2354d388b3eef0215a5a83e P ffebd0cde82845a38c83bbf2c9bf29e1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:48:53.850198 20061 log.cc:1196] T 290c4d09d2354d388b3eef0215a5a83e P ffebd0cde82845a38c83bbf2c9bf29e1: Deleting WAL directory at /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790394221564982-779-0/minicluster-data/ts-2/wal/wals/290c4d09d2354d388b3eef0215a5a83e
I20260926 03:48:53.850792 20061 ts_tablet_manager.cc:1939] T 290c4d09d2354d388b3eef0215a5a83e P ffebd0cde82845a38c83bbf2c9bf29e1: Deleting consensus metadata
I20260926 03:48:53.852833 19557 catalog_manager.cc:4841] TS ffebd0cde82845a38c83bbf2c9bf29e1 (127.0.194.195:38801): tablet 290c4d09d2354d388b3eef0215a5a83e (table test-table [id=52a01d4eacf44f0e8c40580df8535521]) successfully deleted
I20260926 03:48:53.853659 20060 ts_tablet_manager.cc:1918] T 290c4d09d2354d388b3eef0215a5a83e P c405c65ff3fe4685a238d380d3a6e71e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:48:53.853847 20060 log.cc:1196] T 290c4d09d2354d388b3eef0215a5a83e P c405c65ff3fe4685a238d380d3a6e71e: Deleting WAL directory at /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790394221564982-779-0/minicluster-data/ts-1/wal/wals/290c4d09d2354d388b3eef0215a5a83e
I20260926 03:48:53.854393 20060 ts_tablet_manager.cc:1939] T 290c4d09d2354d388b3eef0215a5a83e P c405c65ff3fe4685a238d380d3a6e71e: Deleting consensus metadata
I20260926 03:48:53.856194 19557 catalog_manager.cc:4841] TS c405c65ff3fe4685a238d380d3a6e71e (127.0.194.194:34891): tablet 290c4d09d2354d388b3eef0215a5a83e (table test-table [id=52a01d4eacf44f0e8c40580df8535521]) successfully deleted
I20260926 03:48:53.939301 779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu with pid 19631
I20260926 03:48:53.957574 779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu pid 19762
I20260926 03:48:53.964781 20032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:53.958336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44136 (local address 127.0.194.194:34891)
0926 03:48:53.958549 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0926 03:48:53.958556 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:48:53.958634 (+ 78us) server_negotiation.cc:407] Connection header received
0926 03:48:53.958900 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:48:53.958906 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:48:53.959002 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:48:53.959142 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:48:53.959551 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:48:53.960304 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:48:53.961571 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:48:53.962029 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:48:53.962163 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:48:53.963348 (+ 1185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:48:53.963382 (+ 34us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:48:53.963397 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:48:53.964008 (+ 611us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:48:53.964059 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:48:53.964081 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:48:53.964206 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0926 03:48:53.964309 (+ 103us) server_negotiation.cc:299] Negotiation successful
0926 03:48:53.964419 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260926 03:48:53.965606 19884 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:48:54.078797 779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu with pid 19762
I20260926 03:48:54.097553 779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu pid 19892
I20260926 03:48:54.104234 20036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:54.098386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50846 (local address 127.0.194.195:38801)
0926 03:48:54.098573 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0926 03:48:54.098591 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0926 03:48:54.098616 (+ 25us) server_negotiation.cc:407] Connection header received
0926 03:48:54.098879 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:48:54.098883 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:48:54.098966 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:48:54.099066 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:48:54.099400 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:48:54.100150 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:48:54.101359 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:48:54.101751 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:48:54.101845 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:48:54.103083 (+ 1238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:48:54.103092 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:48:54.103107 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:48:54.103675 (+ 568us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:48:54.103721 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:48:54.103740 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:48:54.103833 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0926 03:48:54.103914 (+ 81us) server_negotiation.cc:299] Negotiation successful
0926 03:48:54.103997 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 03:48:54.104849 20014 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:48:54.192574 779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu with pid 19892
I20260926 03:48:54.209072 779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu pid 19541
I20260926 03:48:54.210227 19602 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:48:54.319211 779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu with pid 19541
2026-09-26T03:48:54Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (2800 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790394221564982-779-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:48:54 dist-test-slave-n2th krb5kdc[20070](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:48:54 dist-test-slave-n2th krb5kdc[20070](info): set up 2 sockets
Sep 26 03:48:54 dist-test-slave-n2th krb5kdc[20070](info): commencing operation
krb5kdc: starting...
W20260926 03:48:55.423089 779 mini_kdc.cc:121] Time spent starting KDC: real 1.063s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790394221564982-779-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790394221564982-779-0/krb5kdc/test-user.keytab.
Sep 26 03:48:55 dist-test-slave-n2th krb5kdc[20070](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394535, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:48:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:48:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.194.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.194.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.194.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790394221564982-779-0/krb5kdc/kudu_127.0.194.254.keytab.
Entry for principal kudu/127.0.194.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790394221564982-779-0/krb5kdc/kudu_127.0.194.254.keytab.
I20260926 03:48:55.683293 779 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
/tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790394221564982-779-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790394221564982-779-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790394221564982-779-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790394221564982-779-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.194.254:34563
--webserver_interface=127.0.194.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790394221564982-779-0/krb5kdc/kudu_127.0.194.254.keytab
--principal=kudu/127.0.194.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.194.212:45979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.194.254:34563
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790394221564982-779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790394221564982-779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790394221564982-779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:48:56.024793 20083 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:48:56.025118 20083 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:48:56.025281 20083 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:48:56.032680 20083 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:48:56.032824 20083 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:48:56.032913 20083 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:48:56.033007 20083 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:48:56.033079 20083 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:48:56.043222 20083 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.194.212:45979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790394221564982-779-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790394221564982-779-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.194.254:34563
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790394221564982-779-0/krb5kdc/kudu_127.0.194.254.keytab
--principal=kudu/127.0.194.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.194.254:34563
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790394221564982-779-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.194.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790394221564982-779-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:48:56.044687 20083 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:48:56.046017 20083 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:48:56.057761 20091 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:48:56.057659 20088 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:48:56.057688 20089 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:48:56.059042 20083 server_base.cc:1034] running on GCE node
I20260926 03:48:56.059823 20083 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:48:56.061251 20083 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:48:56.062467 20083 hybrid_clock.cc:648] HybridClock initialized: now 1790394536062416 us; error 32 us; skew 500 ppm
I20260926 03:48:56.062889 20083 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:48:56 dist-test-slave-n2th krb5kdc[20070](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394536, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:48:56.066166 20083 init.cc:377] Logged in from keytab as kudu/127.0.194.254@KRBTEST.COM (short username kudu)
I20260926 03:48:56.067832 20083 webserver.cc:460] Webserver started at http://127.0.194.254:40529/ using document root <none> and password file <none>
I20260926 03:48:56.068578 20083 fs_manager.cc:359] Metadata directory not provided
I20260926 03:48:56.068727 20083 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:48:56.069051 20083 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:48:56.072261 20083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790394221564982-779-0/minicluster-data/master-0/data/instance:
uuid: "85bd352a016d47f698801ec16930e664"
format_stamp: "Formatted at 2026-09-26 03:48:56 on dist-test-slave-n2th"
server_key: "c8f1689c0e76d7af60f7400c17fedc8e"
server_key_iv: "f54f45b31beb0fa7e6ccca1600bda51b"
server_key_version: "encryptionkey@0"
I20260926 03:48:56.073073 20083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790394221564982-779-0/minicluster-data/master-0/wal/instance:
uuid: "85bd352a016d47f698801ec16930e664"
format_stamp: "Formatted at 2026-09-26 03:48:56 on dist-test-slave-n2th"
server_key: "c8f1689c0e76d7af60f7400c17fedc8e"
server_key_iv: "f54f45b31beb0fa7e6ccca1600bda51b"
server_key_version: "encryptionkey@0"
I20260926 03:48:56.079198 20083 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260926 03:48:56.083127 20098 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:48:56.084663 20083 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 03:48:56.084874 20083 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790394221564982-779-0/minicluster-data/master-0/data,/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790394221564982-779-0/minicluster-data/master-0/wal
uuid: "85bd352a016d47f698801ec16930e664"
format_stamp: "Formatted at 2026-09-26 03:48:56 on dist-test-slave-n2th"
server_key: "c8f1689c0e76d7af60f7400c17fedc8e"
server_key_iv: "f54f45b31beb0fa7e6ccca1600bda51b"
server_key_version: "encryptionkey@0"
I20260926 03:48:56.085099 20083 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790394221564982-779-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790394221564982-779-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790394221564982-779-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:48:56.106889 20083 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:48:56.111263 20083 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:48:56.111585 20083 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:48:56.130952 20083 rpc_server.cc:307] RPC server started. Bound to: 127.0.194.254:34563
I20260926 03:48:56.131075 20149 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.194.254:34563 every 8 connection(s)
I20260926 03:48:56.132427 20083 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790394221564982-779-0/minicluster-data/master-0/data/info.pb
I20260926 03:48:56.136363 20150 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:48:56.138861 779 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu as pid 20083
I20260926 03:48:56.139075 779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790394221564982-779-0/minicluster-data/master-0/wal/instance
I20260926 03:48:56.139533 779 external_mini_cluster.cc:1428] Setting key e2db42b6245cfd854add6a263dd4f6a4
I20260926 03:48:56.149744 20150 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85bd352a016d47f698801ec16930e664: Bootstrap starting.
I20260926 03:48:56.153616 20150 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 85bd352a016d47f698801ec16930e664: Neither blocks nor log segments found. Creating new log.
I20260926 03:48:56.154649 20150 log.cc:824] T 00000000000000000000000000000000 P 85bd352a016d47f698801ec16930e664: Log is configured to *not* fsync() on all Append() calls
Sep 26 03:48:56 dist-test-slave-n2th krb5kdc[20070](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394535, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.194.254@KRBTEST.COM
I20260926 03:48:56.157829 20150 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85bd352a016d47f698801ec16930e664: No bootstrap required, opened a new log
I20260926 03:48:56.164319 20153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:56.141556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37184 (local address 127.0.194.254:34563)
0926 03:48:56.142383 (+ 827us) server_negotiation.cc:206] Beginning negotiation
0926 03:48:56.142409 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0926 03:48:56.142463 (+ 54us) server_negotiation.cc:407] Connection header received
0926 03:48:56.144917 (+ 2454us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:48:56.144948 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:48:56.145526 (+ 578us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:48:56.146014 (+ 488us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:48:56.147484 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:48:56.149464 (+ 1980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:48:56.150940 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:48:56.152102 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:48:56.152280 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:48:56.156673 (+ 4393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:48:56.156682 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:48:56.156698 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:48:56.156740 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:48:56.159989 (+ 3249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:48:56.160743 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:48:56.160750 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:48:56.160757 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:48:56.160930 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:48:56.161343 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:48:56.161347 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:48:56.161349 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:48:56.161916 (+ 567us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:48:56.162253 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0926 03:48:56.162710 (+ 457us) server_negotiation.cc:299] Negotiation successful
0926 03:48:56.163216 (+ 506us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":576,"thread_start_us":322,"threads_started":1}
I20260926 03:48:56.166476 20150 raft_consensus.cc:348] T 00000000000000000000000000000000 P 85bd352a016d47f698801ec16930e664 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85bd352a016d47f698801ec16930e664" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 34563 } }
I20260926 03:48:56.167024 20150 raft_consensus.cc:374] T 00000000000000000000000000000000 P 85bd352a016d47f698801ec16930e664 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:48:56.167145 20150 raft_consensus.cc:729] T 00000000000000000000000000000000 P 85bd352a016d47f698801ec16930e664 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85bd352a016d47f698801ec16930e664, State: Initialized, Role: FOLLOWER
I20260926 03:48:56.167819 20150 consensus_queue.cc:260] T 00000000000000000000000000000000 P 85bd352a016d47f698801ec16930e664 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85bd352a016d47f698801ec16930e664" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 34563 } }
I20260926 03:48:56.168100 20150 raft_consensus.cc:388] T 00000000000000000000000000000000 P 85bd352a016d47f698801ec16930e664 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:48:56.168205 20150 raft_consensus.cc:482] T 00000000000000000000000000000000 P 85bd352a016d47f698801ec16930e664 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:48:56.168329 20150 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 85bd352a016d47f698801ec16930e664 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:48:56.171160 20150 raft_consensus.cc:504] T 00000000000000000000000000000000 P 85bd352a016d47f698801ec16930e664 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85bd352a016d47f698801ec16930e664" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 34563 } }
I20260926 03:48:56.171715 20150 leader_election.cc:302] T 00000000000000000000000000000000 P 85bd352a016d47f698801ec16930e664 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 85bd352a016d47f698801ec16930e664; no voters:
I20260926 03:48:56.172322 20150 leader_election.cc:288] T 00000000000000000000000000000000 P 85bd352a016d47f698801ec16930e664 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:48:56.172473 20155 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 85bd352a016d47f698801ec16930e664 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:48:56.173884 20155 raft_consensus.cc:686] T 00000000000000000000000000000000 P 85bd352a016d47f698801ec16930e664 [term 1 LEADER]: Becoming Leader. State: Replica: 85bd352a016d47f698801ec16930e664, State: Running, Role: LEADER
I20260926 03:48:56.174508 20150 sys_catalog.cc:564] T 00000000000000000000000000000000 P 85bd352a016d47f698801ec16930e664 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:48:56.174453 20155 consensus_queue.cc:237] T 00000000000000000000000000000000 P 85bd352a016d47f698801ec16930e664 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85bd352a016d47f698801ec16930e664" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 34563 } }
I20260926 03:48:56.177376 20156 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85bd352a016d47f698801ec16930e664 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "85bd352a016d47f698801ec16930e664" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85bd352a016d47f698801ec16930e664" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 34563 } } }
I20260926 03:48:56.177838 20156 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85bd352a016d47f698801ec16930e664 [sys.catalog]: This master's current role is: LEADER
I20260926 03:48:56.177718 20157 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85bd352a016d47f698801ec16930e664 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 85bd352a016d47f698801ec16930e664. Latest consensus state: current_term: 1 leader_uuid: "85bd352a016d47f698801ec16930e664" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85bd352a016d47f698801ec16930e664" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 34563 } } }
I20260926 03:48:56.177986 20157 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85bd352a016d47f698801ec16930e664 [sys.catalog]: This master's current role is: LEADER
I20260926 03:48:56.182358 20164 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:48:56.187561 20164 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:48:56.196931 20164 catalog_manager.cc:1347] Generated new cluster ID: 9c987c86900847bc96e40000b34aa7f9
I20260926 03:48:56.197067 20164 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:48:56.209194 20164 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:48:56.210558 20164 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:48:56.220111 20164 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 85bd352a016d47f698801ec16930e664: Generated new TSK 0
I20260926 03:48:56.221037 20164 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.194.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.194.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.194.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790394221564982-779-0/krb5kdc/kudu_127.0.194.193.keytab.
Entry for principal kudu/127.0.194.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790394221564982-779-0/krb5kdc/kudu_127.0.194.193.keytab.
I20260926 03:48:56.295976 779 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
/tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790394221564982-779-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790394221564982-779-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790394221564982-779-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790394221564982-779-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.194.193:0
--local_ip_for_outbound_sockets=127.0.194.193
--webserver_interface=127.0.194.193
--webserver_port=0
--tserver_master_addrs=127.0.194.254:34563
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790394221564982-779-0/krb5kdc/kudu_127.0.194.193.keytab
--principal=kudu/127.0.194.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.194.212:45979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790394221564982-779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790394221564982-779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790394221564982-779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:48:56.665094 20176 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:48:56.665395 20176 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:48:56.665555 20176 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:48:56.673053 20176 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:48:56.673206 20176 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:48:56.673393 20176 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.194.193
I20260926 03:48:56.683997 20176 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.194.212:45979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790394221564982-779-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790394221564982-779-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790394221564982-779-0/krb5kdc/kudu_127.0.194.193.keytab
--principal=kudu/127.0.194.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.194.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790394221564982-779-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.194.193
--webserver_port=0
--tserver_master_addrs=127.0.194.254:34563
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.194.193
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790394221564982-779-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:48:56.685380 20176 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:48:56.686793 20176 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:48:56.700029 20181 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:48:56.700133 20184 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:48:56.700693 20182 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:48:56.701596 20176 server_base.cc:1034] running on GCE node
I20260926 03:48:56.702171 20176 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:48:56.703259 20176 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:48:56.704475 20176 hybrid_clock.cc:648] HybridClock initialized: now 1790394536704428 us; error 42 us; skew 500 ppm
I20260926 03:48:56.704833 20176 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:48:56 dist-test-slave-n2th krb5kdc[20070](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394536, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:48:56.708101 20176 init.cc:377] Logged in from keytab as kudu/127.0.194.193@KRBTEST.COM (short username kudu)
I20260926 03:48:56.709581 20176 webserver.cc:460] Webserver started at http://127.0.194.193:42285/ using document root <none> and password file <none>
I20260926 03:48:56.710290 20176 fs_manager.cc:359] Metadata directory not provided
I20260926 03:48:56.710429 20176 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:48:56.710781 20176 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:48:56.713894 20176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790394221564982-779-0/minicluster-data/ts-0/data/instance:
uuid: "a11bfdc55d3b4a4f8b07dd5a56ce7cbb"
format_stamp: "Formatted at 2026-09-26 03:48:56 on dist-test-slave-n2th"
server_key: "6f4b5d9e88bca838cc2c39000961b5a8"
server_key_iv: "ab4f82f50616362292e6947cf5684be2"
server_key_version: "encryptionkey@0"
I20260926 03:48:56.714710 20176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790394221564982-779-0/minicluster-data/ts-0/wal/instance:
uuid: "a11bfdc55d3b4a4f8b07dd5a56ce7cbb"
format_stamp: "Formatted at 2026-09-26 03:48:56 on dist-test-slave-n2th"
server_key: "6f4b5d9e88bca838cc2c39000961b5a8"
server_key_iv: "ab4f82f50616362292e6947cf5684be2"
server_key_version: "encryptionkey@0"
I20260926 03:48:56.720112 20176 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260926 03:48:56.724090 20191 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:48:56.725509 20176 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 03:48:56.725689 20176 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790394221564982-779-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790394221564982-779-0/minicluster-data/ts-0/wal
uuid: "a11bfdc55d3b4a4f8b07dd5a56ce7cbb"
format_stamp: "Formatted at 2026-09-26 03:48:56 on dist-test-slave-n2th"
server_key: "6f4b5d9e88bca838cc2c39000961b5a8"
server_key_iv: "ab4f82f50616362292e6947cf5684be2"
server_key_version: "encryptionkey@0"
I20260926 03:48:56.725937 20176 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790394221564982-779-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790394221564982-779-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790394221564982-779-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:48:56.739434 20176 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:48:56.744067 20176 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:48:56.744372 20176 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:48:56.745378 20176 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:48:56.746958 20176 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:48:56.747053 20176 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:48:56.747205 20176 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:48:56.747275 20176 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:48:56.780043 20176 rpc_server.cc:307] RPC server started. Bound to: 127.0.194.193:36703
I20260926 03:48:56.780126 20303 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.194.193:36703 every 8 connection(s)
I20260926 03:48:56.781401 20176 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790394221564982-779-0/minicluster-data/ts-0/data/info.pb
I20260926 03:48:56.788015 779 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu as pid 20176
I20260926 03:48:56.788211 779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790394221564982-779-0/minicluster-data/ts-0/wal/instance
I20260926 03:48:56.788717 779 external_mini_cluster.cc:1428] Setting key 456177b4a2968212e606132a234b9f82
Sep 26 03:48:56 dist-test-slave-n2th krb5kdc[20070](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394536, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.193@KRBTEST.COM for kudu/127.0.194.254@KRBTEST.COM
I20260926 03:48:56.802218 20306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:56.784055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.193:46571 (local address 127.0.194.254:34563)
0926 03:48:56.784529 (+ 474us) server_negotiation.cc:206] Beginning negotiation
0926 03:48:56.784533 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:48:56.785374 (+ 841us) server_negotiation.cc:407] Connection header received
0926 03:48:56.787383 (+ 2009us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:48:56.787389 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:48:56.787482 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:48:56.787596 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:48:56.789478 (+ 1882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:48:56.790550 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:48:56.792171 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:48:56.792578 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:48:56.792713 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:48:56.796575 (+ 3862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:48:56.796580 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:48:56.796584 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:48:56.796617 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:48:56.799184 (+ 2567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:48:56.799917 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:48:56.799923 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:48:56.799931 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:48:56.800042 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:48:56.800465 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:48:56.800470 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:48:56.800473 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:48:56.800745 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:48:56.800892 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0926 03:48:56.801608 (+ 716us) server_negotiation.cc:299] Negotiation successful
0926 03:48:56.801850 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":253,"threads_started":1}
I20260926 03:48:56.803320 20307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:56.784523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:34563 (local address 127.0.194.193:46571)
0926 03:48:56.785204 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0926 03:48:56.785252 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 03:48:56.787108 (+ 1856us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:48:56.787846 (+ 738us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:48:56.787854 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:48:56.788595 (+ 741us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:48:56.789215 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:48:56.789252 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:48:56.790710 (+ 1458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:48:56.790715 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:48:56.792019 (+ 1304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:48:56.792031 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:48:56.792817 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:48:56.792825 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:48:56.793067 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:48:56.793781 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:48:56.793808 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:48:56.796321 (+ 2513us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:48:56.799329 (+ 3008us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:48:56.799338 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:48:56.799356 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:48:56.799783 (+ 427us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:48:56.800150 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:48:56.800156 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:48:56.800160 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:48:56.800345 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:48:56.800879 (+ 534us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:48:56.800887 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:48:56.801242 (+ 355us) client_negotiation.cc:769] Sending connection context
0926 03:48:56.801506 (+ 264us) client_negotiation.cc:240] Negotiation successful
0926 03:48:56.802074 (+ 568us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":257,"threads_started":1}
I20260926 03:48:56.804946 20304 heartbeater.cc:347] Connected to a master server at 127.0.194.254:34563
I20260926 03:48:56.805289 20304 heartbeater.cc:464] Registering TS with master...
I20260926 03:48:56.806284 20304 heartbeater.cc:511] Master 127.0.194.254:34563 requested a full tablet report, sending...
I20260926 03:48:56.808920 20115 ts_manager.cc:194] Registered new tserver with Master: a11bfdc55d3b4a4f8b07dd5a56ce7cbb (127.0.194.193:36703)
I20260926 03:48:56.811225 20115 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.194.193@KRBTEST.COM'} at 127.0.194.193:46571
WARNING: no policy specified for kudu/127.0.194.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.194.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.194.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790394221564982-779-0/krb5kdc/kudu_127.0.194.194.keytab.
Entry for principal kudu/127.0.194.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790394221564982-779-0/krb5kdc/kudu_127.0.194.194.keytab.
I20260926 03:48:56.849680 779 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
/tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790394221564982-779-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790394221564982-779-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790394221564982-779-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790394221564982-779-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.194.194:0
--local_ip_for_outbound_sockets=127.0.194.194
--webserver_interface=127.0.194.194
--webserver_port=0
--tserver_master_addrs=127.0.194.254:34563
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790394221564982-779-0/krb5kdc/kudu_127.0.194.194.keytab
--principal=kudu/127.0.194.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.194.212:45979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790394221564982-779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790394221564982-779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790394221564982-779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:48:57.185225 20310 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:48:57.185534 20310 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:48:57.185700 20310 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:48:57.193470 20310 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:48:57.193596 20310 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:48:57.193776 20310 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.194.194
I20260926 03:48:57.203747 20310 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.194.212:45979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790394221564982-779-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790394221564982-779-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790394221564982-779-0/krb5kdc/kudu_127.0.194.194.keytab
--principal=kudu/127.0.194.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.194.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790394221564982-779-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.194.194
--webserver_port=0
--tserver_master_addrs=127.0.194.254:34563
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.194.194
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790394221564982-779-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:48:57.205132 20310 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:48:57.206511 20310 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:48:57.220064 20315 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:48:57.220345 20316 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:48:57.220377 20318 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:48:57.220964 20310 server_base.cc:1034] running on GCE node
I20260926 03:48:57.221618 20310 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:48:57.222609 20310 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:48:57.223901 20310 hybrid_clock.cc:648] HybridClock initialized: now 1790394537223835 us; error 70 us; skew 500 ppm
I20260926 03:48:57.224263 20310 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:48:57 dist-test-slave-n2th krb5kdc[20070](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394537, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:48:57.235407 20310 init.cc:377] Logged in from keytab as kudu/127.0.194.194@KRBTEST.COM (short username kudu)
I20260926 03:48:57.237040 20310 webserver.cc:460] Webserver started at http://127.0.194.194:45463/ using document root <none> and password file <none>
I20260926 03:48:57.237802 20310 fs_manager.cc:359] Metadata directory not provided
I20260926 03:48:57.238096 20310 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:48:57.238471 20310 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:48:57.241654 20310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790394221564982-779-0/minicluster-data/ts-1/data/instance:
uuid: "55f8ba7692c941608e4fede79b6193a4"
format_stamp: "Formatted at 2026-09-26 03:48:57 on dist-test-slave-n2th"
server_key: "f18b92a4f5f09967eeed0ce59fb5a884"
server_key_iv: "bfb9964637bdf40fa5a4177bd8b6fd44"
server_key_version: "encryptionkey@0"
I20260926 03:48:57.242524 20310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790394221564982-779-0/minicluster-data/ts-1/wal/instance:
uuid: "55f8ba7692c941608e4fede79b6193a4"
format_stamp: "Formatted at 2026-09-26 03:48:57 on dist-test-slave-n2th"
server_key: "f18b92a4f5f09967eeed0ce59fb5a884"
server_key_iv: "bfb9964637bdf40fa5a4177bd8b6fd44"
server_key_version: "encryptionkey@0"
I20260926 03:48:57.248302 20310 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260926 03:48:57.252133 20325 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:48:57.253588 20310 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 03:48:57.253782 20310 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790394221564982-779-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790394221564982-779-0/minicluster-data/ts-1/wal
uuid: "55f8ba7692c941608e4fede79b6193a4"
format_stamp: "Formatted at 2026-09-26 03:48:57 on dist-test-slave-n2th"
server_key: "f18b92a4f5f09967eeed0ce59fb5a884"
server_key_iv: "bfb9964637bdf40fa5a4177bd8b6fd44"
server_key_version: "encryptionkey@0"
I20260926 03:48:57.254050 20310 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790394221564982-779-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790394221564982-779-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790394221564982-779-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:48:57.279214 20310 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:48:57.283639 20310 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:48:57.283957 20310 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:48:57.285061 20310 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:48:57.286526 20310 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:48:57.286633 20310 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:48:57.286816 20310 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:48:57.286890 20310 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260926 03:48:57.322080 20310 rpc_server.cc:307] RPC server started. Bound to: 127.0.194.194:45567
I20260926 03:48:57.322124 20437 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.194.194:45567 every 8 connection(s)
I20260926 03:48:57.323499 20310 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790394221564982-779-0/minicluster-data/ts-1/data/info.pb
I20260926 03:48:57.325487 779 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu as pid 20310
I20260926 03:48:57.325695 779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790394221564982-779-0/minicluster-data/ts-1/wal/instance
I20260926 03:48:57.326239 779 external_mini_cluster.cc:1428] Setting key dba1b88edfdab34dc4c726cfb59f82ae
Sep 26 03:48:57 dist-test-slave-n2th krb5kdc[20070](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394537, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.194@KRBTEST.COM for kudu/127.0.194.254@KRBTEST.COM
I20260926 03:48:57.346117 20440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:57.326201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.194:58099 (local address 127.0.194.254:34563)
0926 03:48:57.327143 (+ 942us) server_negotiation.cc:206] Beginning negotiation
0926 03:48:57.327152 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:48:57.327918 (+ 766us) server_negotiation.cc:407] Connection header received
0926 03:48:57.330116 (+ 2198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:48:57.330125 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:48:57.330220 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:48:57.330339 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:48:57.332057 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:48:57.333225 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:48:57.334935 (+ 1710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:48:57.335696 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:48:57.335853 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:48:57.340073 (+ 4220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:48:57.340078 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:48:57.340082 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:48:57.340117 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:48:57.342923 (+ 2806us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:48:57.343731 (+ 808us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:48:57.343735 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:48:57.343738 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:48:57.343872 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:48:57.344314 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:48:57.344318 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:48:57.344322 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:48:57.344596 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:48:57.344750 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0926 03:48:57.345493 (+ 743us) server_negotiation.cc:299] Negotiation successful
0926 03:48:57.345722 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":805,"thread_start_us":697,"threads_started":1}
I20260926 03:48:57.347373 20441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:57.326825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:34563 (local address 127.0.194.194:58099)
0926 03:48:57.327738 (+ 913us) negotiation.cc:107] Waiting for socket to connect
0926 03:48:57.327790 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 03:48:57.329795 (+ 2005us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:48:57.330583 (+ 788us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:48:57.330593 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:48:57.331196 (+ 603us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:48:57.331877 (+ 681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:48:57.331895 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:48:57.333379 (+ 1484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:48:57.333382 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:48:57.334712 (+ 1330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:48:57.334724 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:48:57.335906 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:48:57.335911 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:48:57.336119 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:48:57.336865 (+ 746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:48:57.336887 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:48:57.339793 (+ 2906us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:48:57.343089 (+ 3296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:48:57.343100 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:48:57.343118 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:48:57.343589 (+ 471us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:48:57.343989 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:48:57.343994 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:48:57.343998 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:48:57.344189 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:48:57.344733 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:48:57.344742 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:48:57.345110 (+ 368us) client_negotiation.cc:769] Sending connection context
0926 03:48:57.345382 (+ 272us) client_negotiation.cc:240] Negotiation successful
0926 03:48:57.345944 (+ 562us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":576,"thread_start_us":346,"threads_started":1}
I20260926 03:48:57.349015 20438 heartbeater.cc:347] Connected to a master server at 127.0.194.254:34563
I20260926 03:48:57.349359 20438 heartbeater.cc:464] Registering TS with master...
I20260926 03:48:57.350088 20438 heartbeater.cc:511] Master 127.0.194.254:34563 requested a full tablet report, sending...
I20260926 03:48:57.351835 20115 ts_manager.cc:194] Registered new tserver with Master: 55f8ba7692c941608e4fede79b6193a4 (127.0.194.194:45567)
WARNING: no policy specified for kudu/127.0.194.195@KRBTEST.COM; defaulting to no policy
I20260926 03:48:57.352739 20115 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.194.194@KRBTEST.COM'} at 127.0.194.194:58099
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.194.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.194.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790394221564982-779-0/krb5kdc/kudu_127.0.194.195.keytab.
Entry for principal kudu/127.0.194.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790394221564982-779-0/krb5kdc/kudu_127.0.194.195.keytab.
I20260926 03:48:57.385219 779 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
/tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790394221564982-779-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790394221564982-779-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790394221564982-779-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790394221564982-779-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.194.195:0
--local_ip_for_outbound_sockets=127.0.194.195
--webserver_interface=127.0.194.195
--webserver_port=0
--tserver_master_addrs=127.0.194.254:34563
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790394221564982-779-0/krb5kdc/kudu_127.0.194.195.keytab
--principal=kudu/127.0.194.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.194.212:45979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790394221564982-779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790394221564982-779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790394221564982-779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:48:57.726310 20444 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:48:57.726667 20444 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:48:57.726866 20444 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:48:57.734524 20444 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:48:57.734707 20444 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:48:57.734915 20444 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.194.195
I20260926 03:48:57.745332 20444 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.194.212:45979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790394221564982-779-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790394221564982-779-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790394221564982-779-0/krb5kdc/kudu_127.0.194.195.keytab
--principal=kudu/127.0.194.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.194.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790394221564982-779-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.194.195
--webserver_port=0
--tserver_master_addrs=127.0.194.254:34563
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.194.195
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790394221564982-779-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:48:57.746865 20444 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:48:57.748342 20444 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:48:57.762066 20449 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:48:57.762099 20450 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:48:57.762125 20452 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:48:57.763074 20444 server_base.cc:1034] running on GCE node
I20260926 03:48:57.763823 20444 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:48:57.764858 20444 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:48:57.766095 20444 hybrid_clock.cc:648] HybridClock initialized: now 1790394537766036 us; error 56 us; skew 500 ppm
I20260926 03:48:57.766435 20444 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:48:57 dist-test-slave-n2th krb5kdc[20070](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394537, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:48:57.769662 20444 init.cc:377] Logged in from keytab as kudu/127.0.194.195@KRBTEST.COM (short username kudu)
I20260926 03:48:57.771211 20444 webserver.cc:460] Webserver started at http://127.0.194.195:33659/ using document root <none> and password file <none>
I20260926 03:48:57.771903 20444 fs_manager.cc:359] Metadata directory not provided
I20260926 03:48:57.772007 20444 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:48:57.772261 20444 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:48:57.775336 20444 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790394221564982-779-0/minicluster-data/ts-2/data/instance:
uuid: "7d0cf86f8d304b9395c293923bf453e4"
format_stamp: "Formatted at 2026-09-26 03:48:57 on dist-test-slave-n2th"
server_key: "518dc6a5f5f7ca6eb64730cda8942396"
server_key_iv: "fc2fbada46b50c63ec6c9dfeda3b4e65"
server_key_version: "encryptionkey@0"
I20260926 03:48:57.776175 20444 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790394221564982-779-0/minicluster-data/ts-2/wal/instance:
uuid: "7d0cf86f8d304b9395c293923bf453e4"
format_stamp: "Formatted at 2026-09-26 03:48:57 on dist-test-slave-n2th"
server_key: "518dc6a5f5f7ca6eb64730cda8942396"
server_key_iv: "fc2fbada46b50c63ec6c9dfeda3b4e65"
server_key_version: "encryptionkey@0"
I20260926 03:48:57.781831 20444 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 03:48:57.786034 20459 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:48:57.787606 20444 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260926 03:48:57.787796 20444 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790394221564982-779-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790394221564982-779-0/minicluster-data/ts-2/wal
uuid: "7d0cf86f8d304b9395c293923bf453e4"
format_stamp: "Formatted at 2026-09-26 03:48:57 on dist-test-slave-n2th"
server_key: "518dc6a5f5f7ca6eb64730cda8942396"
server_key_iv: "fc2fbada46b50c63ec6c9dfeda3b4e65"
server_key_version: "encryptionkey@0"
I20260926 03:48:57.788025 20444 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790394221564982-779-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790394221564982-779-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790394221564982-779-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:48:57.809155 20444 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:48:57.813469 20444 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:48:57.813771 20444 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:48:57.814877 20444 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:48:57.815439 20304 heartbeater.cc:503] Master 127.0.194.254:34563 was elected leader, sending a full tablet report...
I20260926 03:48:57.816614 20444 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:48:57.816722 20444 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:48:57.816851 20444 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:48:57.816918 20444 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:48:57.852659 20444 rpc_server.cc:307] RPC server started. Bound to: 127.0.194.195:37165
I20260926 03:48:57.852716 20571 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.194.195:37165 every 8 connection(s)
I20260926 03:48:57.854144 20444 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790394221564982-779-0/minicluster-data/ts-2/data/info.pb
I20260926 03:48:57.861410 779 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu as pid 20444
I20260926 03:48:57.861632 779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790394221564982-779-0/minicluster-data/ts-2/wal/instance
I20260926 03:48:57.862103 779 external_mini_cluster.cc:1428] Setting key 7ba7ec8fdfdde0449c6d1ae782be09bc
Sep 26 03:48:57 dist-test-slave-n2th krb5kdc[20070](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394537, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.195@KRBTEST.COM for kudu/127.0.194.254@KRBTEST.COM
I20260926 03:48:57.876896 20574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:57.857034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.195:35871 (local address 127.0.194.254:34563)
0926 03:48:57.857517 (+ 483us) server_negotiation.cc:206] Beginning negotiation
0926 03:48:57.857522 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:48:57.858501 (+ 979us) server_negotiation.cc:407] Connection header received
0926 03:48:57.860639 (+ 2138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:48:57.860644 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:48:57.860745 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:48:57.860872 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:48:57.862867 (+ 1995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:48:57.863931 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:48:57.866089 (+ 2158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:48:57.866536 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:48:57.866666 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:48:57.870378 (+ 3712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:48:57.870383 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:48:57.870387 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:48:57.870426 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:48:57.873489 (+ 3063us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:48:57.874245 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:48:57.874249 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:48:57.874252 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:48:57.874382 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:48:57.874861 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:48:57.874864 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:48:57.874867 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:48:57.875211 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:48:57.875346 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0926 03:48:57.876225 (+ 879us) server_negotiation.cc:299] Negotiation successful
0926 03:48:57.876472 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":253,"threads_started":1}
I20260926 03:48:57.878280 20575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:57.857503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:34563 (local address 127.0.194.195:35871)
0926 03:48:57.858323 (+ 820us) negotiation.cc:107] Waiting for socket to connect
0926 03:48:57.858363 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 03:48:57.860315 (+ 1952us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:48:57.861165 (+ 850us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:48:57.861174 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:48:57.861866 (+ 692us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:48:57.862617 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:48:57.862645 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:48:57.864099 (+ 1454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:48:57.864103 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:48:57.865914 (+ 1811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:48:57.865930 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:48:57.866870 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:48:57.866877 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:48:57.867093 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:48:57.867792 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:48:57.867815 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:48:57.870134 (+ 2319us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:48:57.873640 (+ 3506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:48:57.873648 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:48:57.873663 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:48:57.874114 (+ 451us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:48:57.874487 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:48:57.874490 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:48:57.874492 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:48:57.874671 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:48:57.875325 (+ 654us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:48:57.875331 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:48:57.875701 (+ 370us) client_negotiation.cc:769] Sending connection context
0926 03:48:57.875997 (+ 296us) client_negotiation.cc:240] Negotiation successful
0926 03:48:57.876574 (+ 577us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":574,"thread_start_us":335,"threads_started":1}
I20260926 03:48:57.879838 20572 heartbeater.cc:347] Connected to a master server at 127.0.194.254:34563
I20260926 03:48:57.880162 20572 heartbeater.cc:464] Registering TS with master...
I20260926 03:48:57.880903 20572 heartbeater.cc:511] Master 127.0.194.254:34563 requested a full tablet report, sending...
I20260926 03:48:57.882421 20115 ts_manager.cc:194] Registered new tserver with Master: 7d0cf86f8d304b9395c293923bf453e4 (127.0.194.195:37165)
I20260926 03:48:57.883340 20115 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.194.195@KRBTEST.COM'} at 127.0.194.195:35871
I20260926 03:48:57.890618 779 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 03:48:57.909761 20574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:57.895524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53792 (local address 127.0.194.254:34563)
0926 03:48:57.895764 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0926 03:48:57.895772 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:48:57.896097 (+ 325us) server_negotiation.cc:407] Connection header received
0926 03:48:57.896321 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:48:57.896325 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:48:57.896395 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:48:57.896504 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:48:57.897920 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:48:57.898685 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:48:57.900034 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:48:57.900437 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:48:57.900581 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:48:57.903073 (+ 2492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:48:57.903078 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:48:57.903082 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:48:57.903120 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:48:57.906129 (+ 3009us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:48:57.907201 (+ 1072us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:48:57.907207 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:48:57.907211 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:48:57.907391 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:48:57.907945 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:48:57.907955 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:48:57.907958 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:48:57.908372 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:48:57.908667 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0926 03:48:57.909095 (+ 428us) server_negotiation.cc:299] Negotiation successful
0926 03:48:57.909389 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 03:48:57.914618 20115 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53792:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 03:48:57.921859 20115 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 03:48:57.951926 20586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:57.935832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.194:45567 (local address 127.0.0.1:43304)
0926 03:48:57.938391 (+ 2559us) negotiation.cc:107] Waiting for socket to connect
0926 03:48:57.938411 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 03:48:57.938563 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:48:57.939785 (+ 1222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:48:57.939798 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:48:57.939841 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:48:57.940383 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:48:57.940399 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:48:57.943011 (+ 2612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:48:57.943020 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:48:57.946912 (+ 3892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:48:57.946940 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:48:57.950370 (+ 3430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:48:57.950381 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:48:57.950713 (+ 332us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:48:57.950814 (+ 101us) client_negotiation.cc:769] Sending connection context
0926 03:48:57.950991 (+ 177us) client_negotiation.cc:240] Negotiation successful
0926 03:48:57.951299 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2407,"mutex_wait_us":5,"thread_start_us":2446,"threads_started":1}
I20260926 03:48:57.952440 20588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:57.936438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43304 (local address 127.0.194.194:45567)
0926 03:48:57.938324 (+ 1886us) server_negotiation.cc:206] Beginning negotiation
0926 03:48:57.938332 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:48:57.938523 (+ 191us) server_negotiation.cc:407] Connection header received
0926 03:48:57.938808 (+ 285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:48:57.938818 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:48:57.939175 (+ 357us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:48:57.939423 (+ 248us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:48:57.940644 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:48:57.942795 (+ 2151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:48:57.947230 (+ 4435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:48:57.949674 (+ 2444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:48:57.949877 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:48:57.950178 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0926 03:48:57.951966 (+ 1788us) server_negotiation.cc:299] Negotiation successful
0926 03:48:57.952195 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1739,"thread_start_us":1639,"threads_started":1}
I20260926 03:48:57.953974 20585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:57.935580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.195:37165 (local address 127.0.0.1:57076)
0926 03:48:57.937046 (+ 1466us) negotiation.cc:107] Waiting for socket to connect
0926 03:48:57.937102 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0926 03:48:57.937328 (+ 226us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:48:57.940581 (+ 3253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:48:57.940589 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:48:57.940625 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:48:57.941134 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:48:57.941148 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:48:57.945435 (+ 4287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:48:57.945458 (+ 23us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:48:57.950307 (+ 4849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:48:57.950332 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:48:57.953084 (+ 2752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:48:57.953092 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:48:57.953287 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:48:57.953322 (+ 35us) client_negotiation.cc:769] Sending connection context
0926 03:48:57.953407 (+ 85us) client_negotiation.cc:240] Negotiation successful
0926 03:48:57.953565 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1096,"thread_start_us":916,"threads_started":1}
I20260926 03:48:57.954686 20587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:57.935879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57076 (local address 127.0.194.195:37165)
0926 03:48:57.938063 (+ 2184us) server_negotiation.cc:206] Beginning negotiation
0926 03:48:57.938076 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 03:48:57.938116 (+ 40us) server_negotiation.cc:407] Connection header received
0926 03:48:57.938220 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:48:57.938228 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:48:57.939927 (+ 1699us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:48:57.940181 (+ 254us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:48:57.941420 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:48:57.945197 (+ 3777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:48:57.950597 (+ 5400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:48:57.952207 (+ 1610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:48:57.952630 (+ 423us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:48:57.953900 (+ 1270us) server_negotiation.cc:1035] Waiting for connection context
0926 03:48:57.953985 (+ 85us) server_negotiation.cc:299] Negotiation successful
0926 03:48:57.954194 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2000,"thread_start_us":1835,"threads_started":1}
I20260926 03:48:57.957010 20373 tablet_service.cc:1467] Processing CreateTablet for tablet d2d96936b1474def8433c75ebc3174ad (DEFAULT_TABLE table=test-table [id=9aa2c26ee16b46bf9f201397f0c29442]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:48:57.958607 20373 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2d96936b1474def8433c75ebc3174ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:48:57.958971 20507 tablet_service.cc:1467] Processing CreateTablet for tablet d2d96936b1474def8433c75ebc3174ad (DEFAULT_TABLE table=test-table [id=9aa2c26ee16b46bf9f201397f0c29442]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:48:57.960613 20507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2d96936b1474def8433c75ebc3174ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:48:57.961711 20589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:57.938677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.193:36703 (local address 127.0.0.1:51944)
0926 03:48:57.951969 (+ 13292us) negotiation.cc:107] Waiting for socket to connect
0926 03:48:57.951991 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 03:48:57.952427 (+ 436us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:48:57.953529 (+ 1102us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:48:57.953534 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:48:57.953562 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:48:57.954003 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:48:57.954015 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:48:57.956153 (+ 2138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:48:57.956158 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:48:57.958373 (+ 2215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:48:57.958400 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:48:57.960847 (+ 2447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:48:57.960853 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:48:57.961048 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:48:57.961086 (+ 38us) client_negotiation.cc:769] Sending connection context
0926 03:48:57.961202 (+ 116us) client_negotiation.cc:240] Negotiation successful
0926 03:48:57.961367 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13141,"thread_start_us":12968,"threads_started":1}
I20260926 03:48:57.962062 20590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:57.939026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51944 (local address 127.0.194.193:36703)
0926 03:48:57.951265 (+ 12239us) server_negotiation.cc:206] Beginning negotiation
0926 03:48:57.951274 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:48:57.952285 (+ 1011us) server_negotiation.cc:407] Connection header received
0926 03:48:57.952662 (+ 377us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:48:57.952672 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:48:57.953004 (+ 332us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:48:57.953202 (+ 198us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:48:57.954188 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:48:57.955969 (+ 1781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:48:57.958646 (+ 2677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:48:57.960231 (+ 1585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:48:57.960378 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:48:57.960613 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0926 03:48:57.961286 (+ 673us) server_negotiation.cc:299] Negotiation successful
0926 03:48:57.961499 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12031,"thread_start_us":12243,"threads_started":1}
I20260926 03:48:57.966522 20239 tablet_service.cc:1467] Processing CreateTablet for tablet d2d96936b1474def8433c75ebc3174ad (DEFAULT_TABLE table=test-table [id=9aa2c26ee16b46bf9f201397f0c29442]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:48:57.968334 20239 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2d96936b1474def8433c75ebc3174ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:48:57.971834 20591 tablet_bootstrap.cc:492] T d2d96936b1474def8433c75ebc3174ad P 55f8ba7692c941608e4fede79b6193a4: Bootstrap starting.
I20260926 03:48:57.975337 20591 tablet_bootstrap.cc:654] T d2d96936b1474def8433c75ebc3174ad P 55f8ba7692c941608e4fede79b6193a4: Neither blocks nor log segments found. Creating new log.
I20260926 03:48:57.976565 20591 log.cc:824] T d2d96936b1474def8433c75ebc3174ad P 55f8ba7692c941608e4fede79b6193a4: Log is configured to *not* fsync() on all Append() calls
I20260926 03:48:57.977746 20592 tablet_bootstrap.cc:492] T d2d96936b1474def8433c75ebc3174ad P 7d0cf86f8d304b9395c293923bf453e4: Bootstrap starting.
I20260926 03:48:57.981600 20592 tablet_bootstrap.cc:654] T d2d96936b1474def8433c75ebc3174ad P 7d0cf86f8d304b9395c293923bf453e4: Neither blocks nor log segments found. Creating new log.
I20260926 03:48:57.982194 20591 tablet_bootstrap.cc:492] T d2d96936b1474def8433c75ebc3174ad P 55f8ba7692c941608e4fede79b6193a4: No bootstrap required, opened a new log
I20260926 03:48:57.982502 20591 ts_tablet_manager.cc:1397] T d2d96936b1474def8433c75ebc3174ad P 55f8ba7692c941608e4fede79b6193a4: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260926 03:48:57.983017 20592 log.cc:824] T d2d96936b1474def8433c75ebc3174ad P 7d0cf86f8d304b9395c293923bf453e4: Log is configured to *not* fsync() on all Append() calls
I20260926 03:48:57.985970 20592 tablet_bootstrap.cc:492] T d2d96936b1474def8433c75ebc3174ad P 7d0cf86f8d304b9395c293923bf453e4: No bootstrap required, opened a new log
I20260926 03:48:57.986325 20592 ts_tablet_manager.cc:1397] T d2d96936b1474def8433c75ebc3174ad P 7d0cf86f8d304b9395c293923bf453e4: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260926 03:48:57.986963 20595 tablet_bootstrap.cc:492] T d2d96936b1474def8433c75ebc3174ad P a11bfdc55d3b4a4f8b07dd5a56ce7cbb: Bootstrap starting.
I20260926 03:48:57.991189 20595 tablet_bootstrap.cc:654] T d2d96936b1474def8433c75ebc3174ad P a11bfdc55d3b4a4f8b07dd5a56ce7cbb: Neither blocks nor log segments found. Creating new log.
I20260926 03:48:57.992733 20595 log.cc:824] T d2d96936b1474def8433c75ebc3174ad P a11bfdc55d3b4a4f8b07dd5a56ce7cbb: Log is configured to *not* fsync() on all Append() calls
I20260926 03:48:57.993633 20591 raft_consensus.cc:348] T d2d96936b1474def8433c75ebc3174ad P 55f8ba7692c941608e4fede79b6193a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d0cf86f8d304b9395c293923bf453e4" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 37165 } } peers { permanent_uuid: "55f8ba7692c941608e4fede79b6193a4" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 45567 } } peers { permanent_uuid: "a11bfdc55d3b4a4f8b07dd5a56ce7cbb" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 36703 } }
I20260926 03:48:57.994189 20591 raft_consensus.cc:374] T d2d96936b1474def8433c75ebc3174ad P 55f8ba7692c941608e4fede79b6193a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:48:57.994315 20591 raft_consensus.cc:729] T d2d96936b1474def8433c75ebc3174ad P 55f8ba7692c941608e4fede79b6193a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55f8ba7692c941608e4fede79b6193a4, State: Initialized, Role: FOLLOWER
I20260926 03:48:57.995092 20591 consensus_queue.cc:260] T d2d96936b1474def8433c75ebc3174ad P 55f8ba7692c941608e4fede79b6193a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d0cf86f8d304b9395c293923bf453e4" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 37165 } } peers { permanent_uuid: "55f8ba7692c941608e4fede79b6193a4" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 45567 } } peers { permanent_uuid: "a11bfdc55d3b4a4f8b07dd5a56ce7cbb" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 36703 } }
I20260926 03:48:57.996256 20438 heartbeater.cc:503] Master 127.0.194.254:34563 was elected leader, sending a full tablet report...
I20260926 03:48:57.996527 20595 tablet_bootstrap.cc:492] T d2d96936b1474def8433c75ebc3174ad P a11bfdc55d3b4a4f8b07dd5a56ce7cbb: No bootstrap required, opened a new log
I20260926 03:48:57.996835 20595 ts_tablet_manager.cc:1397] T d2d96936b1474def8433c75ebc3174ad P a11bfdc55d3b4a4f8b07dd5a56ce7cbb: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260926 03:48:57.997583 20592 raft_consensus.cc:348] T d2d96936b1474def8433c75ebc3174ad P 7d0cf86f8d304b9395c293923bf453e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d0cf86f8d304b9395c293923bf453e4" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 37165 } } peers { permanent_uuid: "55f8ba7692c941608e4fede79b6193a4" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 45567 } } peers { permanent_uuid: "a11bfdc55d3b4a4f8b07dd5a56ce7cbb" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 36703 } }
I20260926 03:48:57.998078 20592 raft_consensus.cc:374] T d2d96936b1474def8433c75ebc3174ad P 7d0cf86f8d304b9395c293923bf453e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:48:57.998183 20592 raft_consensus.cc:729] T d2d96936b1474def8433c75ebc3174ad P 7d0cf86f8d304b9395c293923bf453e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d0cf86f8d304b9395c293923bf453e4, State: Initialized, Role: FOLLOWER
I20260926 03:48:57.998422 20591 ts_tablet_manager.cc:1428] T d2d96936b1474def8433c75ebc3174ad P 55f8ba7692c941608e4fede79b6193a4: Time spent starting tablet: real 0.016s user 0.016s sys 0.000s
I20260926 03:48:57.999212 20592 consensus_queue.cc:260] T d2d96936b1474def8433c75ebc3174ad P 7d0cf86f8d304b9395c293923bf453e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d0cf86f8d304b9395c293923bf453e4" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 37165 } } peers { permanent_uuid: "55f8ba7692c941608e4fede79b6193a4" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 45567 } } peers { permanent_uuid: "a11bfdc55d3b4a4f8b07dd5a56ce7cbb" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 36703 } }
I20260926 03:48:58.000474 20572 heartbeater.cc:503] Master 127.0.194.254:34563 was elected leader, sending a full tablet report...
I20260926 03:48:58.001184 20592 ts_tablet_manager.cc:1428] T d2d96936b1474def8433c75ebc3174ad P 7d0cf86f8d304b9395c293923bf453e4: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
I20260926 03:48:58.008332 20595 raft_consensus.cc:348] T d2d96936b1474def8433c75ebc3174ad P a11bfdc55d3b4a4f8b07dd5a56ce7cbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d0cf86f8d304b9395c293923bf453e4" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 37165 } } peers { permanent_uuid: "55f8ba7692c941608e4fede79b6193a4" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 45567 } } peers { permanent_uuid: "a11bfdc55d3b4a4f8b07dd5a56ce7cbb" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 36703 } }
I20260926 03:48:58.008764 20595 raft_consensus.cc:374] T d2d96936b1474def8433c75ebc3174ad P a11bfdc55d3b4a4f8b07dd5a56ce7cbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:48:58.008901 20595 raft_consensus.cc:729] T d2d96936b1474def8433c75ebc3174ad P a11bfdc55d3b4a4f8b07dd5a56ce7cbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a11bfdc55d3b4a4f8b07dd5a56ce7cbb, State: Initialized, Role: FOLLOWER
I20260926 03:48:58.009573 20595 consensus_queue.cc:260] T d2d96936b1474def8433c75ebc3174ad P a11bfdc55d3b4a4f8b07dd5a56ce7cbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d0cf86f8d304b9395c293923bf453e4" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 37165 } } peers { permanent_uuid: "55f8ba7692c941608e4fede79b6193a4" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 45567 } } peers { permanent_uuid: "a11bfdc55d3b4a4f8b07dd5a56ce7cbb" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 36703 } }
I20260926 03:48:58.011162 20595 ts_tablet_manager.cc:1428] T d2d96936b1474def8433c75ebc3174ad P a11bfdc55d3b4a4f8b07dd5a56ce7cbb: Time spent starting tablet: real 0.014s user 0.011s sys 0.004s
W20260926 03:48:58.034031 20305 tablet.cc:2369] T d2d96936b1474def8433c75ebc3174ad P a11bfdc55d3b4a4f8b07dd5a56ce7cbb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 03:48:58.075762 20439 tablet.cc:2369] T d2d96936b1474def8433c75ebc3174ad P 55f8ba7692c941608e4fede79b6193a4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 03:48:58.098222 20599 raft_consensus.cc:482] T d2d96936b1474def8433c75ebc3174ad P a11bfdc55d3b4a4f8b07dd5a56ce7cbb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 03:48:58.098574 20599 raft_consensus.cc:504] T d2d96936b1474def8433c75ebc3174ad P a11bfdc55d3b4a4f8b07dd5a56ce7cbb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d0cf86f8d304b9395c293923bf453e4" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 37165 } } peers { permanent_uuid: "55f8ba7692c941608e4fede79b6193a4" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 45567 } } peers { permanent_uuid: "a11bfdc55d3b4a4f8b07dd5a56ce7cbb" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 36703 } }
I20260926 03:48:58.100468 20599 leader_election.cc:288] T d2d96936b1474def8433c75ebc3174ad P a11bfdc55d3b4a4f8b07dd5a56ce7cbb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7d0cf86f8d304b9395c293923bf453e4 (127.0.194.195:37165), 55f8ba7692c941608e4fede79b6193a4 (127.0.194.194:45567)
W20260926 03:48:58.106020 20573 tablet.cc:2369] T d2d96936b1474def8433c75ebc3174ad P 7d0cf86f8d304b9395c293923bf453e4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 03:48:58.108974 20587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:58.100895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.193:49355 (local address 127.0.194.195:37165)
0926 03:48:58.101163 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0926 03:48:58.101172 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:48:58.102325 (+ 1153us) server_negotiation.cc:407] Connection header received
0926 03:48:58.102499 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:48:58.102503 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:48:58.102564 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:48:58.102673 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:48:58.103449 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:48:58.104223 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:48:58.107048 (+ 2825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:48:58.107913 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:48:58.108137 (+ 224us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:48:58.108224 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0926 03:48:58.108556 (+ 332us) server_negotiation.cc:299] Negotiation successful
0926 03:48:58.108692 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260926 03:48:58.109127 20588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:58.101282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.193:37265 (local address 127.0.194.194:45567)
0926 03:48:58.101475 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0926 03:48:58.101483 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:48:58.102191 (+ 708us) server_negotiation.cc:407] Connection header received
0926 03:48:58.102465 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:48:58.102473 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:48:58.102549 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:48:58.102619 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:48:58.103320 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:48:58.104081 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:48:58.107089 (+ 3008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:48:58.108145 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:48:58.108358 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:48:58.108436 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 03:48:58.108728 (+ 292us) server_negotiation.cc:299] Negotiation successful
0926 03:48:58.108861 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 03:48:58.109167 20600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:58.100978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.195:37165 (local address 127.0.194.193:49355)
0926 03:48:58.102168 (+ 1190us) negotiation.cc:107] Waiting for socket to connect
0926 03:48:58.102187 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 03:48:58.102328 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:48:58.102710 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:48:58.102715 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:48:58.102763 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:48:58.103276 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:48:58.103289 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:48:58.104439 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:48:58.104444 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:48:58.106852 (+ 2408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:48:58.106869 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:48:58.108164 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:48:58.108171 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:48:58.108374 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:48:58.108399 (+ 25us) client_negotiation.cc:769] Sending connection context
0926 03:48:58.108518 (+ 119us) client_negotiation.cc:240] Negotiation successful
0926 03:48:58.108669 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1065,"mutex_wait_us":44,"spinlock_wait_cycles":28928,"thread_start_us":500,"threads_started":1}
I20260926 03:48:58.109290 20601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:58.100750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.194:45567 (local address 127.0.194.193:37265)
0926 03:48:58.102056 (+ 1306us) negotiation.cc:107] Waiting for socket to connect
0926 03:48:58.102073 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 03:48:58.102292 (+ 219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:48:58.102668 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:48:58.102672 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:48:58.102697 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:48:58.103152 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:48:58.103162 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:48:58.104226 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:48:58.104232 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:48:58.106920 (+ 2688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:48:58.106937 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:48:58.108394 (+ 1457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:48:58.108400 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:48:58.108568 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:48:58.108592 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 03:48:58.108700 (+ 108us) client_negotiation.cc:240] Negotiation successful
0926 03:48:58.108855 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1155,"mutex_wait_us":82,"thread_start_us":1022,"threads_started":1}
I20260926 03:48:58.110525 20393 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2d96936b1474def8433c75ebc3174ad" candidate_uuid: "a11bfdc55d3b4a4f8b07dd5a56ce7cbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55f8ba7692c941608e4fede79b6193a4" is_pre_election: true
I20260926 03:48:58.110884 20527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2d96936b1474def8433c75ebc3174ad" candidate_uuid: "a11bfdc55d3b4a4f8b07dd5a56ce7cbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d0cf86f8d304b9395c293923bf453e4" is_pre_election: true
I20260926 03:48:58.111110 20393 raft_consensus.cc:2454] T d2d96936b1474def8433c75ebc3174ad P 55f8ba7692c941608e4fede79b6193a4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a11bfdc55d3b4a4f8b07dd5a56ce7cbb in term 0.
I20260926 03:48:58.111420 20527 raft_consensus.cc:2454] T d2d96936b1474def8433c75ebc3174ad P 7d0cf86f8d304b9395c293923bf453e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a11bfdc55d3b4a4f8b07dd5a56ce7cbb in term 0.
I20260926 03:48:58.112046 20193 leader_election.cc:302] T d2d96936b1474def8433c75ebc3174ad P a11bfdc55d3b4a4f8b07dd5a56ce7cbb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 55f8ba7692c941608e4fede79b6193a4, a11bfdc55d3b4a4f8b07dd5a56ce7cbb; no voters:
I20260926 03:48:58.112603 20599 raft_consensus.cc:2781] T d2d96936b1474def8433c75ebc3174ad P a11bfdc55d3b4a4f8b07dd5a56ce7cbb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 03:48:58.112780 20599 raft_consensus.cc:482] T d2d96936b1474def8433c75ebc3174ad P a11bfdc55d3b4a4f8b07dd5a56ce7cbb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 03:48:58.112918 20599 raft_consensus.cc:3037] T d2d96936b1474def8433c75ebc3174ad P a11bfdc55d3b4a4f8b07dd5a56ce7cbb [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:48:58.115561 20599 raft_consensus.cc:504] T d2d96936b1474def8433c75ebc3174ad P a11bfdc55d3b4a4f8b07dd5a56ce7cbb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d0cf86f8d304b9395c293923bf453e4" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 37165 } } peers { permanent_uuid: "55f8ba7692c941608e4fede79b6193a4" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 45567 } } peers { permanent_uuid: "a11bfdc55d3b4a4f8b07dd5a56ce7cbb" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 36703 } }
I20260926 03:48:58.116346 20599 leader_election.cc:288] T d2d96936b1474def8433c75ebc3174ad P a11bfdc55d3b4a4f8b07dd5a56ce7cbb [CANDIDATE]: Term 1 election: Requested vote from peers 7d0cf86f8d304b9395c293923bf453e4 (127.0.194.195:37165), 55f8ba7692c941608e4fede79b6193a4 (127.0.194.194:45567)
I20260926 03:48:58.117116 20527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2d96936b1474def8433c75ebc3174ad" candidate_uuid: "a11bfdc55d3b4a4f8b07dd5a56ce7cbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d0cf86f8d304b9395c293923bf453e4"
I20260926 03:48:58.117121 20393 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2d96936b1474def8433c75ebc3174ad" candidate_uuid: "a11bfdc55d3b4a4f8b07dd5a56ce7cbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55f8ba7692c941608e4fede79b6193a4"
I20260926 03:48:58.117367 20527 raft_consensus.cc:3037] T d2d96936b1474def8433c75ebc3174ad P 7d0cf86f8d304b9395c293923bf453e4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:48:58.117393 20393 raft_consensus.cc:3037] T d2d96936b1474def8433c75ebc3174ad P 55f8ba7692c941608e4fede79b6193a4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:48:58.120098 20527 raft_consensus.cc:2454] T d2d96936b1474def8433c75ebc3174ad P 7d0cf86f8d304b9395c293923bf453e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a11bfdc55d3b4a4f8b07dd5a56ce7cbb in term 1.
I20260926 03:48:58.120095 20393 raft_consensus.cc:2454] T d2d96936b1474def8433c75ebc3174ad P 55f8ba7692c941608e4fede79b6193a4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a11bfdc55d3b4a4f8b07dd5a56ce7cbb in term 1.
I20260926 03:48:58.120707 20193 leader_election.cc:302] T d2d96936b1474def8433c75ebc3174ad P a11bfdc55d3b4a4f8b07dd5a56ce7cbb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 55f8ba7692c941608e4fede79b6193a4, a11bfdc55d3b4a4f8b07dd5a56ce7cbb; no voters:
I20260926 03:48:58.121150 20599 raft_consensus.cc:2781] T d2d96936b1474def8433c75ebc3174ad P a11bfdc55d3b4a4f8b07dd5a56ce7cbb [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:48:58.121752 20599 raft_consensus.cc:686] T d2d96936b1474def8433c75ebc3174ad P a11bfdc55d3b4a4f8b07dd5a56ce7cbb [term 1 LEADER]: Becoming Leader. State: Replica: a11bfdc55d3b4a4f8b07dd5a56ce7cbb, State: Running, Role: LEADER
I20260926 03:48:58.122325 20599 consensus_queue.cc:237] T d2d96936b1474def8433c75ebc3174ad P a11bfdc55d3b4a4f8b07dd5a56ce7cbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d0cf86f8d304b9395c293923bf453e4" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 37165 } } peers { permanent_uuid: "55f8ba7692c941608e4fede79b6193a4" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 45567 } } peers { permanent_uuid: "a11bfdc55d3b4a4f8b07dd5a56ce7cbb" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 36703 } }
I20260926 03:48:58.128022 20115 catalog_manager.cc:5495] T d2d96936b1474def8433c75ebc3174ad P a11bfdc55d3b4a4f8b07dd5a56ce7cbb reported cstate change: term changed from 0 to 1, leader changed from <none> to a11bfdc55d3b4a4f8b07dd5a56ce7cbb (127.0.194.193). New cstate: current_term: 1 leader_uuid: "a11bfdc55d3b4a4f8b07dd5a56ce7cbb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d0cf86f8d304b9395c293923bf453e4" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 37165 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "55f8ba7692c941608e4fede79b6193a4" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 45567 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a11bfdc55d3b4a4f8b07dd5a56ce7cbb" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 36703 } health_report { overall_health: HEALTHY } } }
I20260926 03:48:58.147154 20590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:58.140562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51960 (local address 127.0.194.193:36703)
0926 03:48:58.140791 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0926 03:48:58.140799 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:48:58.140821 (+ 22us) server_negotiation.cc:407] Connection header received
0926 03:48:58.140945 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:48:58.140951 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:48:58.141053 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:48:58.141223 (+ 170us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:48:58.141604 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:48:58.142903 (+ 1299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:48:58.144354 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:48:58.145072 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:48:58.145215 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:48:58.145731 (+ 516us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:48:58.146480 (+ 749us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:48:58.146644 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0926 03:48:58.146700 (+ 56us) server_negotiation.cc:299] Negotiation successful
0926 03:48:58.146846 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 03:48:58.156293 20393 raft_consensus.cc:1261] T d2d96936b1474def8433c75ebc3174ad P 55f8ba7692c941608e4fede79b6193a4 [term 1 FOLLOWER]: Refusing update from remote peer a11bfdc55d3b4a4f8b07dd5a56ce7cbb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:48:58.156296 20527 raft_consensus.cc:1261] T d2d96936b1474def8433c75ebc3174ad P 7d0cf86f8d304b9395c293923bf453e4 [term 1 FOLLOWER]: Refusing update from remote peer a11bfdc55d3b4a4f8b07dd5a56ce7cbb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:48:58.157569 20602 consensus_queue.cc:1035] T d2d96936b1474def8433c75ebc3174ad P a11bfdc55d3b4a4f8b07dd5a56ce7cbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d0cf86f8d304b9395c293923bf453e4" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 37165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:48:58.158016 20599 consensus_queue.cc:1035] T d2d96936b1474def8433c75ebc3174ad P a11bfdc55d3b4a4f8b07dd5a56ce7cbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "55f8ba7692c941608e4fede79b6193a4" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 45567 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:48:58.194847 20114 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53792:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:48:58.195237 20114 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53792:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:48:58.199680 20114 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 85bd352a016d47f698801ec16930e664: Sending DeleteTablet for 3 replicas of tablet d2d96936b1474def8433c75ebc3174ad
I20260926 03:48:58.201018 20373 tablet_service.cc:1514] Processing DeleteTablet for tablet d2d96936b1474def8433c75ebc3174ad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:48:58 UTC) from {username='kudu', principal='kudu/127.0.194.254@KRBTEST.COM'} at 127.0.0.1:43304
I20260926 03:48:58.201057 20507 tablet_service.cc:1514] Processing DeleteTablet for tablet d2d96936b1474def8433c75ebc3174ad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:48:58 UTC) from {username='kudu', principal='kudu/127.0.194.254@KRBTEST.COM'} at 127.0.0.1:57076
I20260926 03:48:58.201928 20615 tablet_replica.cc:330] T d2d96936b1474def8433c75ebc3174ad P 55f8ba7692c941608e4fede79b6193a4: stopping tablet replica
I20260926 03:48:58.202297 20239 tablet_service.cc:1514] Processing DeleteTablet for tablet d2d96936b1474def8433c75ebc3174ad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:48:58 UTC) from {username='kudu', principal='kudu/127.0.194.254@KRBTEST.COM'} at 127.0.0.1:51944
I20260926 03:48:58.202538 20615 raft_consensus.cc:2229] T d2d96936b1474def8433c75ebc3174ad P 55f8ba7692c941608e4fede79b6193a4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:48:58.203151 20615 raft_consensus.cc:2258] T d2d96936b1474def8433c75ebc3174ad P 55f8ba7692c941608e4fede79b6193a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:48:58.204496 779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu pid 20176
I20260926 03:48:58.209890 20615 ts_tablet_manager.cc:1905] T d2d96936b1474def8433c75ebc3174ad P 55f8ba7692c941608e4fede79b6193a4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:48:58.212817 20617 tablet_replica.cc:330] T d2d96936b1474def8433c75ebc3174ad P a11bfdc55d3b4a4f8b07dd5a56ce7cbb: stopping tablet replica
I20260926 03:48:58.213340 20617 raft_consensus.cc:2229] T d2d96936b1474def8433c75ebc3174ad P a11bfdc55d3b4a4f8b07dd5a56ce7cbb [term 1 LEADER]: Raft consensus shutting down.
I20260926 03:48:58.214155 20617 raft_consensus.cc:2258] T d2d96936b1474def8433c75ebc3174ad P a11bfdc55d3b4a4f8b07dd5a56ce7cbb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:48:58.215188 20616 tablet_replica.cc:330] T d2d96936b1474def8433c75ebc3174ad P 7d0cf86f8d304b9395c293923bf453e4: stopping tablet replica
I20260926 03:48:58.215687 20616 raft_consensus.cc:2229] T d2d96936b1474def8433c75ebc3174ad P 7d0cf86f8d304b9395c293923bf453e4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:48:58.216233 20616 raft_consensus.cc:2258] T d2d96936b1474def8433c75ebc3174ad P 7d0cf86f8d304b9395c293923bf453e4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:48:58.218020 20617 ts_tablet_manager.cc:1905] T d2d96936b1474def8433c75ebc3174ad P a11bfdc55d3b4a4f8b07dd5a56ce7cbb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:48:58.218997 20615 ts_tablet_manager.cc:1918] T d2d96936b1474def8433c75ebc3174ad P 55f8ba7692c941608e4fede79b6193a4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:48:58.219218 20615 log.cc:1196] T d2d96936b1474def8433c75ebc3174ad P 55f8ba7692c941608e4fede79b6193a4: Deleting WAL directory at /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790394221564982-779-0/minicluster-data/ts-1/wal/wals/d2d96936b1474def8433c75ebc3174ad
I20260926 03:48:58.219832 20615 ts_tablet_manager.cc:1939] T d2d96936b1474def8433c75ebc3174ad P 55f8ba7692c941608e4fede79b6193a4: Deleting consensus metadata
I20260926 03:48:58.221822 20100 catalog_manager.cc:4841] TS 55f8ba7692c941608e4fede79b6193a4 (127.0.194.194:45567): tablet d2d96936b1474def8433c75ebc3174ad (table test-table [id=9aa2c26ee16b46bf9f201397f0c29442]) successfully deleted
I20260926 03:48:58.225600 20617 ts_tablet_manager.cc:1918] T d2d96936b1474def8433c75ebc3174ad P a11bfdc55d3b4a4f8b07dd5a56ce7cbb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:48:58.225736 20616 ts_tablet_manager.cc:1905] T d2d96936b1474def8433c75ebc3174ad P 7d0cf86f8d304b9395c293923bf453e4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:48:58.225813 20617 log.cc:1196] T d2d96936b1474def8433c75ebc3174ad P a11bfdc55d3b4a4f8b07dd5a56ce7cbb: Deleting WAL directory at /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790394221564982-779-0/minicluster-data/ts-0/wal/wals/d2d96936b1474def8433c75ebc3174ad
I20260926 03:48:58.226395 20617 ts_tablet_manager.cc:1939] T d2d96936b1474def8433c75ebc3174ad P a11bfdc55d3b4a4f8b07dd5a56ce7cbb: Deleting consensus metadata
I20260926 03:48:58.228389 20100 catalog_manager.cc:4841] TS a11bfdc55d3b4a4f8b07dd5a56ce7cbb (127.0.194.193:36703): tablet d2d96936b1474def8433c75ebc3174ad (table test-table [id=9aa2c26ee16b46bf9f201397f0c29442]) successfully deleted
Sep 26 03:48:58 dist-test-slave-n2th krb5kdc[20070](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394535, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.194.193@KRBTEST.COM
I20260926 03:48:58.233727 20616 ts_tablet_manager.cc:1918] T d2d96936b1474def8433c75ebc3174ad P 7d0cf86f8d304b9395c293923bf453e4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:48:58.233937 20616 log.cc:1196] T d2d96936b1474def8433c75ebc3174ad P 7d0cf86f8d304b9395c293923bf453e4: Deleting WAL directory at /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790394221564982-779-0/minicluster-data/ts-2/wal/wals/d2d96936b1474def8433c75ebc3174ad
I20260926 03:48:58.234549 20616 ts_tablet_manager.cc:1939] T d2d96936b1474def8433c75ebc3174ad P 7d0cf86f8d304b9395c293923bf453e4: Deleting consensus metadata
I20260926 03:48:58.236801 20099 catalog_manager.cc:4841] TS 7d0cf86f8d304b9395c293923bf453e4 (127.0.194.195:37165): tablet d2d96936b1474def8433c75ebc3174ad (table test-table [id=9aa2c26ee16b46bf9f201397f0c29442]) successfully deleted
I20260926 03:48:58.240463 20590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:58.205711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51972 (local address 127.0.194.193:36703)
0926 03:48:58.205940 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0926 03:48:58.205951 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 03:48:58.220864 (+ 14913us) server_negotiation.cc:407] Connection header received
0926 03:48:58.221411 (+ 547us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:48:58.221419 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:48:58.221539 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:48:58.221634 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:48:58.223724 (+ 2090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:48:58.225033 (+ 1309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:48:58.227336 (+ 2303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:48:58.228044 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:48:58.228150 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:48:58.233323 (+ 5173us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:48:58.233338 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:48:58.233353 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:48:58.233396 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:48:58.236877 (+ 3481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:48:58.237990 (+ 1113us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:48:58.238001 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:48:58.238004 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:48:58.238179 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:48:58.238722 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:48:58.238727 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:48:58.238729 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:48:58.239175 (+ 446us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:48:58.239424 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0926 03:48:58.239899 (+ 475us) server_negotiation.cc:299] Negotiation successful
0926 03:48:58.240092 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 03:48:58.241297 20299 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:48:58.354882 779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu with pid 20176
I20260926 03:48:58.372722 779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu pid 20310
Sep 26 03:48:58 dist-test-slave-n2th krb5kdc[20070](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394535, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.194.194@KRBTEST.COM
I20260926 03:48:58.387998 20588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:58.373520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43306 (local address 127.0.194.194:45567)
0926 03:48:58.373706 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0926 03:48:58.373711 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:48:58.373729 (+ 18us) server_negotiation.cc:407] Connection header received
0926 03:48:58.373888 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:48:58.373894 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:48:58.373973 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:48:58.374058 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:48:58.375457 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:48:58.376197 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:48:58.377537 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:48:58.377925 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:48:58.378024 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:48:58.382209 (+ 4185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:48:58.382224 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:48:58.382241 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:48:58.382276 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:48:58.385382 (+ 3106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:48:58.386092 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:48:58.386100 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:48:58.386102 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:48:58.386229 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:48:58.386529 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:48:58.386533 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:48:58.386535 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:48:58.386980 (+ 445us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:48:58.387199 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0926 03:48:58.387451 (+ 252us) server_negotiation.cc:299] Negotiation successful
0926 03:48:58.387613 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 03:48:58.388703 20433 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:48:58.477900 779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu with pid 20310
I20260926 03:48:58.494722 779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu pid 20444
Sep 26 03:48:58 dist-test-slave-n2th krb5kdc[20070](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394535, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.194.195@KRBTEST.COM
I20260926 03:48:58.510224 20587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:58.495584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57086 (local address 127.0.194.195:37165)
0926 03:48:58.495764 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0926 03:48:58.495772 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:48:58.495832 (+ 60us) server_negotiation.cc:407] Connection header received
0926 03:48:58.496054 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:48:58.496060 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:48:58.496136 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:48:58.496242 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:48:58.497524 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:48:58.498264 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:48:58.499736 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:48:58.500155 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:48:58.500295 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:48:58.504855 (+ 4560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:48:58.504871 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:48:58.504887 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:48:58.504936 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:48:58.507640 (+ 2704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:48:58.508284 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:48:58.508291 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:48:58.508293 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:48:58.508461 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:48:58.508894 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:48:58.508897 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:48:58.508899 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:48:58.509246 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:48:58.509456 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0926 03:48:58.509769 (+ 313us) server_negotiation.cc:299] Negotiation successful
0926 03:48:58.509964 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 03:48:58.510931 20567 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:48:58.604086 779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu with pid 20444
I20260926 03:48:58.621461 779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu pid 20083
I20260926 03:48:58.622607 20145 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:48:58.733636 779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu with pid 20083
2026-09-26T03:48:58Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4415 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-26T03:48:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:48:58Z Disabled control of system clock
I20260926 03:48:58.812074 779 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
/tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790394221564982-779-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790394221564982-779-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790394221564982-779-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790394221564982-779-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.194.254:38491
--webserver_interface=127.0.194.254
--webserver_port=0
--builtin_ntp_servers=127.0.194.212:46611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.194.254:38491
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 03:48:59.150218 20626 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:48:59.150540 20626 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:48:59.150692 20626 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:48:59.158352 20626 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:48:59.158479 20626 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:48:59.158550 20626 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:48:59.158634 20626 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:48:59.158694 20626 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:48:59.168339 20626 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.194.212:46611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790394221564982-779-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790394221564982-779-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.194.254:38491
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.194.254:38491
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790394221564982-779-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.194.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790394221564982-779-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:48:59.169674 20626 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:48:59.171571 20626 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:48:59.183643 20631 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:48:59.183594 20632 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:48:59.183614 20634 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:48:59.184381 20626 server_base.cc:1034] running on GCE node
I20260926 03:48:59.185220 20626 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:48:59.186667 20626 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:48:59.187942 20626 hybrid_clock.cc:648] HybridClock initialized: now 1790394539187896 us; error 38 us; skew 500 ppm
I20260926 03:48:59.188311 20626 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 03:48:59.190789 20626 webserver.cc:460] Webserver started at http://127.0.194.254:33535/ using document root <none> and password file <none>
I20260926 03:48:59.191605 20626 fs_manager.cc:359] Metadata directory not provided
I20260926 03:48:59.191752 20626 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:48:59.192103 20626 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:48:59.195549 20626 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790394221564982-779-0/minicluster-data/master-0/data/instance:
uuid: "8d96663531b746fa8a0c6454f1c4dc4b"
format_stamp: "Formatted at 2026-09-26 03:48:59 on dist-test-slave-n2th"
server_key: "93f6e1ba72c2641ba3fead41e4c1951d"
server_key_iv: "d2e346268fec3768327b7cd3146eed74"
server_key_version: "encryptionkey@0"
I20260926 03:48:59.196458 20626 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790394221564982-779-0/minicluster-data/master-0/wal/instance:
uuid: "8d96663531b746fa8a0c6454f1c4dc4b"
format_stamp: "Formatted at 2026-09-26 03:48:59 on dist-test-slave-n2th"
server_key: "93f6e1ba72c2641ba3fead41e4c1951d"
server_key_iv: "d2e346268fec3768327b7cd3146eed74"
server_key_version: "encryptionkey@0"
I20260926 03:48:59.202168 20626 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260926 03:48:59.205916 20640 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:48:59.207579 20626 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 03:48:59.207796 20626 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790394221564982-779-0/minicluster-data/master-0/data,/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790394221564982-779-0/minicluster-data/master-0/wal
uuid: "8d96663531b746fa8a0c6454f1c4dc4b"
format_stamp: "Formatted at 2026-09-26 03:48:59 on dist-test-slave-n2th"
server_key: "93f6e1ba72c2641ba3fead41e4c1951d"
server_key_iv: "d2e346268fec3768327b7cd3146eed74"
server_key_version: "encryptionkey@0"
I20260926 03:48:59.208038 20626 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790394221564982-779-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790394221564982-779-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790394221564982-779-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:48:59.243700 20626 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:48:59.244647 20626 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:48:59.244977 20626 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:48:59.264675 20626 rpc_server.cc:307] RPC server started. Bound to: 127.0.194.254:38491
I20260926 03:48:59.264714 20691 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.194.254:38491 every 8 connection(s)
I20260926 03:48:59.266089 20626 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790394221564982-779-0/minicluster-data/master-0/data/info.pb
I20260926 03:48:59.267858 779 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu as pid 20626
I20260926 03:48:59.268064 779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790394221564982-779-0/minicluster-data/master-0/wal/instance
I20260926 03:48:59.268548 779 external_mini_cluster.cc:1428] Setting key b9dccb9058e84e3189d4876bceebbf37
I20260926 03:48:59.270792 20692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:48:59.282065 20695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:59.270623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47306 (local address 127.0.194.254:38491)
0926 03:48:59.271646 (+ 1023us) server_negotiation.cc:206] Beginning negotiation
0926 03:48:59.271658 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 03:48:59.271701 (+ 43us) server_negotiation.cc:407] Connection header received
0926 03:48:59.273370 (+ 1669us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:48:59.273399 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:48:59.273902 (+ 503us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:48:59.274356 (+ 454us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:48:59.274958 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:48:59.276278 (+ 1320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:48:59.277628 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:48:59.278291 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:48:59.278484 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:48:59.279722 (+ 1238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:48:59.279730 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:48:59.279745 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:48:59.280333 (+ 588us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:48:59.280410 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:48:59.280431 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:48:59.280584 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0926 03:48:59.280836 (+ 252us) server_negotiation.cc:299] Negotiation successful
0926 03:48:59.281126 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":592,"thread_start_us":334,"threads_started":1}
I20260926 03:48:59.282804 20692 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d96663531b746fa8a0c6454f1c4dc4b: Bootstrap starting.
I20260926 03:48:59.286191 20692 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8d96663531b746fa8a0c6454f1c4dc4b: Neither blocks nor log segments found. Creating new log.
I20260926 03:48:59.287215 20692 log.cc:824] T 00000000000000000000000000000000 P 8d96663531b746fa8a0c6454f1c4dc4b: Log is configured to *not* fsync() on all Append() calls
I20260926 03:48:59.290035 20692 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d96663531b746fa8a0c6454f1c4dc4b: No bootstrap required, opened a new log
I20260926 03:48:59.297398 20692 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8d96663531b746fa8a0c6454f1c4dc4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d96663531b746fa8a0c6454f1c4dc4b" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 38491 } }
I20260926 03:48:59.297998 20692 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8d96663531b746fa8a0c6454f1c4dc4b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:48:59.298091 20692 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8d96663531b746fa8a0c6454f1c4dc4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d96663531b746fa8a0c6454f1c4dc4b, State: Initialized, Role: FOLLOWER
I20260926 03:48:59.298645 20692 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8d96663531b746fa8a0c6454f1c4dc4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d96663531b746fa8a0c6454f1c4dc4b" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 38491 } }
I20260926 03:48:59.298940 20692 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8d96663531b746fa8a0c6454f1c4dc4b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:48:59.299054 20692 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8d96663531b746fa8a0c6454f1c4dc4b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:48:59.299185 20692 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8d96663531b746fa8a0c6454f1c4dc4b [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:48:59.301363 20692 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8d96663531b746fa8a0c6454f1c4dc4b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d96663531b746fa8a0c6454f1c4dc4b" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 38491 } }
I20260926 03:48:59.301939 20692 leader_election.cc:302] T 00000000000000000000000000000000 P 8d96663531b746fa8a0c6454f1c4dc4b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8d96663531b746fa8a0c6454f1c4dc4b; no voters:
I20260926 03:48:59.302564 20692 leader_election.cc:288] T 00000000000000000000000000000000 P 8d96663531b746fa8a0c6454f1c4dc4b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:48:59.302706 20697 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8d96663531b746fa8a0c6454f1c4dc4b [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:48:59.303480 20697 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8d96663531b746fa8a0c6454f1c4dc4b [term 1 LEADER]: Becoming Leader. State: Replica: 8d96663531b746fa8a0c6454f1c4dc4b, State: Running, Role: LEADER
I20260926 03:48:59.303993 20697 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8d96663531b746fa8a0c6454f1c4dc4b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d96663531b746fa8a0c6454f1c4dc4b" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 38491 } }
I20260926 03:48:59.304751 20692 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8d96663531b746fa8a0c6454f1c4dc4b [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:48:59.306915 20698 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d96663531b746fa8a0c6454f1c4dc4b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8d96663531b746fa8a0c6454f1c4dc4b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d96663531b746fa8a0c6454f1c4dc4b" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 38491 } } }
I20260926 03:48:59.306936 20699 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d96663531b746fa8a0c6454f1c4dc4b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8d96663531b746fa8a0c6454f1c4dc4b. Latest consensus state: current_term: 1 leader_uuid: "8d96663531b746fa8a0c6454f1c4dc4b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d96663531b746fa8a0c6454f1c4dc4b" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 38491 } } }
I20260926 03:48:59.307420 20699 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d96663531b746fa8a0c6454f1c4dc4b [sys.catalog]: This master's current role is: LEADER
I20260926 03:48:59.308064 20698 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d96663531b746fa8a0c6454f1c4dc4b [sys.catalog]: This master's current role is: LEADER
I20260926 03:48:59.309033 20703 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:48:59.315331 20703 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:48:59.324748 20703 catalog_manager.cc:1347] Generated new cluster ID: 2cfc51ac43464ebcaf8cad9aa2b6624f
I20260926 03:48:59.324913 20703 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:48:59.335031 20703 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:48:59.336417 20703 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:48:59.346889 20703 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8d96663531b746fa8a0c6454f1c4dc4b: Generated new TSK 0
I20260926 03:48:59.347798 20703 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 03:48:59.356443 779 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
/tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790394221564982-779-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790394221564982-779-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790394221564982-779-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790394221564982-779-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.194.193:0
--local_ip_for_outbound_sockets=127.0.194.193
--webserver_interface=127.0.194.193
--webserver_port=0
--tserver_master_addrs=127.0.194.254:38491
--builtin_ntp_servers=127.0.194.212:46611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 03:48:59.719857 20716 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:48:59.720322 20716 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:48:59.720484 20716 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:48:59.729228 20716 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:48:59.729390 20716 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:48:59.729573 20716 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.194.193
I20260926 03:48:59.739193 20716 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.194.212:46611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790394221564982-779-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790394221564982-779-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.194.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790394221564982-779-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.194.193
--webserver_port=0
--tserver_master_addrs=127.0.194.254:38491
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.194.193
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790394221564982-779-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:48:59.740556 20716 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:48:59.742411 20716 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:48:59.755898 20721 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:48:59.755867 20722 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:48:59.755959 20724 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:48:59.756842 20716 server_base.cc:1034] running on GCE node
I20260926 03:48:59.757485 20716 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:48:59.758502 20716 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:48:59.759783 20716 hybrid_clock.cc:648] HybridClock initialized: now 1790394539759668 us; error 80 us; skew 500 ppm
I20260926 03:48:59.760178 20716 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 03:48:59.762457 20716 webserver.cc:460] Webserver started at http://127.0.194.193:35013/ using document root <none> and password file <none>
I20260926 03:48:59.763238 20716 fs_manager.cc:359] Metadata directory not provided
I20260926 03:48:59.763366 20716 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:48:59.763700 20716 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:48:59.766652 20716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790394221564982-779-0/minicluster-data/ts-0/data/instance:
uuid: "5a391051f0254261bae36e232509e650"
format_stamp: "Formatted at 2026-09-26 03:48:59 on dist-test-slave-n2th"
server_key: "7a02f2eca37b7ad60b8b5aa3ec062a39"
server_key_iv: "07399a3b02ae566d24f1b0bbfe61ecb2"
server_key_version: "encryptionkey@0"
I20260926 03:48:59.767566 20716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790394221564982-779-0/minicluster-data/ts-0/wal/instance:
uuid: "5a391051f0254261bae36e232509e650"
format_stamp: "Formatted at 2026-09-26 03:48:59 on dist-test-slave-n2th"
server_key: "7a02f2eca37b7ad60b8b5aa3ec062a39"
server_key_iv: "07399a3b02ae566d24f1b0bbfe61ecb2"
server_key_version: "encryptionkey@0"
I20260926 03:48:59.772892 20716 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 03:48:59.776543 20730 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:48:59.778272 20716 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 03:48:59.778461 20716 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790394221564982-779-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790394221564982-779-0/minicluster-data/ts-0/wal
uuid: "5a391051f0254261bae36e232509e650"
format_stamp: "Formatted at 2026-09-26 03:48:59 on dist-test-slave-n2th"
server_key: "7a02f2eca37b7ad60b8b5aa3ec062a39"
server_key_iv: "07399a3b02ae566d24f1b0bbfe61ecb2"
server_key_version: "encryptionkey@0"
I20260926 03:48:59.778699 20716 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790394221564982-779-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790394221564982-779-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790394221564982-779-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:48:59.797113 20716 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:48:59.797999 20716 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:48:59.798326 20716 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:48:59.799391 20716 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:48:59.800905 20716 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:48:59.801038 20716 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:48:59.801188 20716 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:48:59.801280 20716 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:48:59.835276 20716 rpc_server.cc:307] RPC server started. Bound to: 127.0.194.193:36963
I20260926 03:48:59.835325 20842 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.194.193:36963 every 8 connection(s)
I20260926 03:48:59.836700 20716 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790394221564982-779-0/minicluster-data/ts-0/data/info.pb
I20260926 03:48:59.847231 779 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu as pid 20716
I20260926 03:48:59.847453 779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790394221564982-779-0/minicluster-data/ts-0/wal/instance
I20260926 03:48:59.847990 779 external_mini_cluster.cc:1428] Setting key 5028d8c6895150fc21a17089c62c0013
I20260926 03:48:59.851687 20845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:59.839415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.193:40369 (local address 127.0.194.254:38491)
0926 03:48:59.839841 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0926 03:48:59.839846 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:48:59.841163 (+ 1317us) server_negotiation.cc:407] Connection header received
0926 03:48:59.843105 (+ 1942us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:48:59.843111 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:48:59.843209 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:48:59.843335 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:48:59.844503 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:48:59.845317 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:48:59.847371 (+ 2054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:48:59.847986 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:48:59.848135 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:48:59.850608 (+ 2473us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:48:59.850615 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:48:59.850619 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:48:59.850644 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:48:59.850697 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:48:59.850716 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:48:59.850853 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0926 03:48:59.851211 (+ 358us) server_negotiation.cc:299] Negotiation successful
0926 03:48:59.851330 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":217,"threads_started":1}
I20260926 03:48:59.852123 779 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
/tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790394221564982-779-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790394221564982-779-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790394221564982-779-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790394221564982-779-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.194.194:0
--local_ip_for_outbound_sockets=127.0.194.194
--webserver_interface=127.0.194.194
--webserver_port=0
--tserver_master_addrs=127.0.194.254:38491
--builtin_ntp_servers=127.0.194.212:46611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 03:48:59.853130 20846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:48:59.840087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:38491 (local address 127.0.194.193:40369)
0926 03:48:59.840991 (+ 904us) negotiation.cc:107] Waiting for socket to connect
0926 03:48:59.841035 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 03:48:59.842798 (+ 1763us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:48:59.843604 (+ 806us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:48:59.843614 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:48:59.843716 (+ 102us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:48:59.844317 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:48:59.844335 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:48:59.845496 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:48:59.845501 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:48:59.847177 (+ 1676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:48:59.847195 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:48:59.848458 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:48:59.848466 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:48:59.848804 (+ 338us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:48:59.850219 (+ 1415us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 03:48:59.850223 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:48:59.850290 (+ 67us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 03:48:59.850390 (+ 100us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:48:59.850858 (+ 468us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:48:59.850870 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:48:59.850879 (+ 9us) client_negotiation.cc:769] Sending connection context
0926 03:48:59.851111 (+ 232us) client_negotiation.cc:240] Negotiation successful
0926 03:48:59.851549 (+ 438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":271,"threads_started":1}
I20260926 03:48:59.855026 20843 heartbeater.cc:347] Connected to a master server at 127.0.194.254:38491
I20260926 03:48:59.855372 20843 heartbeater.cc:464] Registering TS with master...
I20260926 03:48:59.856222 20843 heartbeater.cc:511] Master 127.0.194.254:38491 requested a full tablet report, sending...
I20260926 03:48:59.858439 20657 ts_manager.cc:194] Registered new tserver with Master: 5a391051f0254261bae36e232509e650 (127.0.194.193:36963)
I20260926 03:48:59.860009 20657 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.194.193:40369
W20260926 03:49:00.193408 20847 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:49:00.193750 20847 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:49:00.193914 20847 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:49:00.201539 20847 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:49:00.201704 20847 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:49:00.201890 20847 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.194.194
I20260926 03:49:00.212028 20847 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.194.212:46611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790394221564982-779-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790394221564982-779-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.194.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790394221564982-779-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.194.194
--webserver_port=0
--tserver_master_addrs=127.0.194.254:38491
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.194.194
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790394221564982-779-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:49:00.213470 20847 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:49:00.215420 20847 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:49:00.228919 20852 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:49:00.228976 20853 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:49:00.228976 20855 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:49:00.229784 20847 server_base.cc:1034] running on GCE node
I20260926 03:49:00.230393 20847 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:49:00.231453 20847 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:49:00.232705 20847 hybrid_clock.cc:648] HybridClock initialized: now 1790394540232651 us; error 51 us; skew 500 ppm
I20260926 03:49:00.233088 20847 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 03:49:00.235576 20847 webserver.cc:460] Webserver started at http://127.0.194.194:40679/ using document root <none> and password file <none>
I20260926 03:49:00.236327 20847 fs_manager.cc:359] Metadata directory not provided
I20260926 03:49:00.236464 20847 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:49:00.236801 20847 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:49:00.239988 20847 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790394221564982-779-0/minicluster-data/ts-1/data/instance:
uuid: "ea3819de22564d2fb659e8827119e270"
format_stamp: "Formatted at 2026-09-26 03:49:00 on dist-test-slave-n2th"
server_key: "e110b5db8c06c52d4b1996a10bbb9a16"
server_key_iv: "87d3d9e641966814861e8f98af5e010b"
server_key_version: "encryptionkey@0"
I20260926 03:49:00.240833 20847 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790394221564982-779-0/minicluster-data/ts-1/wal/instance:
uuid: "ea3819de22564d2fb659e8827119e270"
format_stamp: "Formatted at 2026-09-26 03:49:00 on dist-test-slave-n2th"
server_key: "e110b5db8c06c52d4b1996a10bbb9a16"
server_key_iv: "87d3d9e641966814861e8f98af5e010b"
server_key_version: "encryptionkey@0"
I20260926 03:49:00.246435 20847 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 03:49:00.250332 20861 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:49:00.251902 20847 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 03:49:00.252061 20847 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790394221564982-779-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790394221564982-779-0/minicluster-data/ts-1/wal
uuid: "ea3819de22564d2fb659e8827119e270"
format_stamp: "Formatted at 2026-09-26 03:49:00 on dist-test-slave-n2th"
server_key: "e110b5db8c06c52d4b1996a10bbb9a16"
server_key_iv: "87d3d9e641966814861e8f98af5e010b"
server_key_version: "encryptionkey@0"
I20260926 03:49:00.252262 20847 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790394221564982-779-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790394221564982-779-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790394221564982-779-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:49:00.282336 20847 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:49:00.283408 20847 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:49:00.283759 20847 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:49:00.284798 20847 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:49:00.286337 20847 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:49:00.286466 20847 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:49:00.286597 20847 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:49:00.286690 20847 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:49:00.321581 20847 rpc_server.cc:307] RPC server started. Bound to: 127.0.194.194:44431
I20260926 03:49:00.321640 20973 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.194.194:44431 every 8 connection(s)
I20260926 03:49:00.323172 20847 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790394221564982-779-0/minicluster-data/ts-1/data/info.pb
I20260926 03:49:00.329361 779 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu as pid 20847
I20260926 03:49:00.329568 779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790394221564982-779-0/minicluster-data/ts-1/wal/instance
I20260926 03:49:00.330108 779 external_mini_cluster.cc:1428] Setting key cb3a9ff1a62cef076133bc8b2191b03c
I20260926 03:49:00.333238 779 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
/tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790394221564982-779-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790394221564982-779-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790394221564982-779-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790394221564982-779-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.194.195:0
--local_ip_for_outbound_sockets=127.0.194.195
--webserver_interface=127.0.194.195
--webserver_port=0
--tserver_master_addrs=127.0.194.254:38491
--builtin_ntp_servers=127.0.194.212:46611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 03:49:00.337499 20845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:49:00.325991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.194:56731 (local address 127.0.194.254:38491)
0926 03:49:00.326169 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0926 03:49:00.326174 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:49:00.327770 (+ 1596us) server_negotiation.cc:407] Connection header received
0926 03:49:00.330220 (+ 2450us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:49:00.330226 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:49:00.330328 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:49:00.330417 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:49:00.331896 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:49:00.332814 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:49:00.334557 (+ 1743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:49:00.335060 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:49:00.335203 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:49:00.336675 (+ 1472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:49:00.336680 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:49:00.336684 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:49:00.336704 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:49:00.336736 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:49:00.336748 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:49:00.336848 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0926 03:49:00.337086 (+ 238us) server_negotiation.cc:299] Negotiation successful
0926 03:49:00.337194 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 03:49:00.338828 20976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:49:00.326473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:38491 (local address 127.0.194.194:56731)
0926 03:49:00.327587 (+ 1114us) negotiation.cc:107] Waiting for socket to connect
0926 03:49:00.327632 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 03:49:00.329803 (+ 2171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:49:00.330814 (+ 1011us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:49:00.330832 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:49:00.330943 (+ 111us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:49:00.331711 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:49:00.331735 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:49:00.332972 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:49:00.332977 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:49:00.334390 (+ 1413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:49:00.334403 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:49:00.335305 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:49:00.335312 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:49:00.335558 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:49:00.336304 (+ 746us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 03:49:00.336307 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:49:00.336356 (+ 49us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 03:49:00.336444 (+ 88us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:49:00.336845 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:49:00.336858 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:49:00.336869 (+ 11us) client_negotiation.cc:769] Sending connection context
0926 03:49:00.337077 (+ 208us) client_negotiation.cc:240] Negotiation successful
0926 03:49:00.337532 (+ 455us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":613,"thread_start_us":424,"threads_started":1}
I20260926 03:49:00.340405 20974 heartbeater.cc:347] Connected to a master server at 127.0.194.254:38491
I20260926 03:49:00.340744 20974 heartbeater.cc:464] Registering TS with master...
I20260926 03:49:00.341534 20974 heartbeater.cc:511] Master 127.0.194.254:38491 requested a full tablet report, sending...
I20260926 03:49:00.343048 20657 ts_manager.cc:194] Registered new tserver with Master: ea3819de22564d2fb659e8827119e270 (127.0.194.194:44431)
I20260926 03:49:00.343895 20657 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.194.194:56731
W20260926 03:49:00.672977 20977 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:49:00.673301 20977 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:49:00.673388 20977 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:49:00.681008 20977 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:49:00.681120 20977 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:49:00.681228 20977 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.194.195
I20260926 03:49:00.691205 20977 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.194.212:46611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790394221564982-779-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790394221564982-779-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.194.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790394221564982-779-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.194.195
--webserver_port=0
--tserver_master_addrs=127.0.194.254:38491
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.194.195
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790394221564982-779-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:49:00.692548 20977 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:49:00.694446 20977 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:49:00.709218 20982 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:49:00.709167 20985 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:49:00.709264 20983 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:49:00.710445 20977 server_base.cc:1034] running on GCE node
I20260926 03:49:00.711284 20977 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:49:00.712541 20977 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:49:00.713760 20977 hybrid_clock.cc:648] HybridClock initialized: now 1790394540713694 us; error 73 us; skew 500 ppm
I20260926 03:49:00.714130 20977 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 03:49:00.716724 20977 webserver.cc:460] Webserver started at http://127.0.194.195:44581/ using document root <none> and password file <none>
I20260926 03:49:00.717491 20977 fs_manager.cc:359] Metadata directory not provided
I20260926 03:49:00.717618 20977 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:49:00.717984 20977 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:49:00.721247 20977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790394221564982-779-0/minicluster-data/ts-2/data/instance:
uuid: "fe64bd86a31547909778a43df40ade18"
format_stamp: "Formatted at 2026-09-26 03:49:00 on dist-test-slave-n2th"
server_key: "cb602f13d36473980a64d5e04d6b5c8f"
server_key_iv: "c4bd064cfe11aeb19f67f3b7c05e9654"
server_key_version: "encryptionkey@0"
I20260926 03:49:00.722131 20977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790394221564982-779-0/minicluster-data/ts-2/wal/instance:
uuid: "fe64bd86a31547909778a43df40ade18"
format_stamp: "Formatted at 2026-09-26 03:49:00 on dist-test-slave-n2th"
server_key: "cb602f13d36473980a64d5e04d6b5c8f"
server_key_iv: "c4bd064cfe11aeb19f67f3b7c05e9654"
server_key_version: "encryptionkey@0"
I20260926 03:49:00.727720 20977 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260926 03:49:00.731544 20991 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:49:00.732954 20977 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 03:49:00.733162 20977 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790394221564982-779-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790394221564982-779-0/minicluster-data/ts-2/wal
uuid: "fe64bd86a31547909778a43df40ade18"
format_stamp: "Formatted at 2026-09-26 03:49:00 on dist-test-slave-n2th"
server_key: "cb602f13d36473980a64d5e04d6b5c8f"
server_key_iv: "c4bd064cfe11aeb19f67f3b7c05e9654"
server_key_version: "encryptionkey@0"
I20260926 03:49:00.733409 20977 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790394221564982-779-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790394221564982-779-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790394221564982-779-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:49:00.757369 20977 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:49:00.758267 20977 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:49:00.758589 20977 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:49:00.759621 20977 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:49:00.761197 20977 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:49:00.761309 20977 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:49:00.761439 20977 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:49:00.761528 20977 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:49:00.796175 20977 rpc_server.cc:307] RPC server started. Bound to: 127.0.194.195:45721
I20260926 03:49:00.796277 21103 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.194.195:45721 every 8 connection(s)
I20260926 03:49:00.797541 20977 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790394221564982-779-0/minicluster-data/ts-2/data/info.pb
I20260926 03:49:00.800341 779 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu as pid 20977
I20260926 03:49:00.800535 779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790394221564982-779-0/minicluster-data/ts-2/wal/instance
I20260926 03:49:00.801126 779 external_mini_cluster.cc:1428] Setting key e14a0539f94e59b2204effca674176a5
I20260926 03:49:00.811466 20845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:49:00.800544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.195:41625 (local address 127.0.194.254:38491)
0926 03:49:00.800767 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0926 03:49:00.800771 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:49:00.802441 (+ 1670us) server_negotiation.cc:407] Connection header received
0926 03:49:00.804761 (+ 2320us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:49:00.804766 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:49:00.804857 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:49:00.804982 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:49:00.806172 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:49:00.807000 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:49:00.808589 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:49:00.809025 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:49:00.809164 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:49:00.810532 (+ 1368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:49:00.810537 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:49:00.810540 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:49:00.810559 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:49:00.810590 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:49:00.810602 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:49:00.810699 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0926 03:49:00.811011 (+ 312us) server_negotiation.cc:299] Negotiation successful
0926 03:49:00.811137 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260926 03:49:00.812668 21106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:49:00.801127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:38491 (local address 127.0.194.195:41625)
0926 03:49:00.802265 (+ 1138us) negotiation.cc:107] Waiting for socket to connect
0926 03:49:00.802315 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 03:49:00.804445 (+ 2130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:49:00.805257 (+ 812us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:49:00.805266 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:49:00.805371 (+ 105us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:49:00.805960 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:49:00.805977 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:49:00.807158 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:49:00.807163 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:49:00.808430 (+ 1267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:49:00.808442 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:49:00.809259 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:49:00.809267 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:49:00.809509 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:49:00.810196 (+ 687us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 03:49:00.810199 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:49:00.810240 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 03:49:00.810329 (+ 89us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:49:00.810697 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:49:00.810711 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:49:00.810722 (+ 11us) client_negotiation.cc:769] Sending connection context
0926 03:49:00.810971 (+ 249us) client_negotiation.cc:240] Negotiation successful
0926 03:49:00.811345 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":569,"thread_start_us":315,"threads_started":1}
I20260926 03:49:00.814182 21104 heartbeater.cc:347] Connected to a master server at 127.0.194.254:38491
I20260926 03:49:00.814524 21104 heartbeater.cc:464] Registering TS with master...
I20260926 03:49:00.815469 21104 heartbeater.cc:511] Master 127.0.194.254:38491 requested a full tablet report, sending...
I20260926 03:49:00.817142 20657 ts_manager.cc:194] Registered new tserver with Master: fe64bd86a31547909778a43df40ade18 (127.0.194.195:45721)
I20260926 03:49:00.817946 20657 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.194.195:41625
I20260926 03:49:00.829555 779 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 03:49:00.840457 20845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:49:00.833804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47320 (local address 127.0.194.254:38491)
0926 03:49:00.834013 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0926 03:49:00.834017 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:49:00.834586 (+ 569us) server_negotiation.cc:407] Connection header received
0926 03:49:00.834836 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:49:00.834846 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:49:00.834925 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:49:00.835037 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:49:00.835556 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:49:00.836329 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:49:00.837833 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:49:00.838364 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:49:00.838509 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:49:00.839712 (+ 1203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:49:00.839718 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:49:00.839730 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:49:00.839756 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:49:00.839790 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:49:00.839802 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:49:00.839900 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 03:49:00.840049 (+ 149us) server_negotiation.cc:299] Negotiation successful
0926 03:49:00.840151 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 03:49:00.844904 20657 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47320:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 03:49:00.852831 20657 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 03:49:00.863452 20843 heartbeater.cc:503] Master 127.0.194.254:38491 was elected leader, sending a full tablet report...
I20260926 03:49:00.879889 21117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:49:00.867061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.194:44431 (local address 127.0.0.1:59492)
0926 03:49:00.867818 (+ 757us) negotiation.cc:107] Waiting for socket to connect
0926 03:49:00.867865 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 03:49:00.868050 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:49:00.868666 (+ 616us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:49:00.868673 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:49:00.868700 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:49:00.869068 (+ 368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:49:00.869084 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:49:00.871539 (+ 2455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:49:00.871548 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:49:00.874790 (+ 3242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:49:00.874812 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:49:00.878346 (+ 3534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:49:00.878356 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:49:00.878707 (+ 351us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:49:00.878807 (+ 100us) client_negotiation.cc:769] Sending connection context
0926 03:49:00.878990 (+ 183us) client_negotiation.cc:240] Negotiation successful
0926 03:49:00.879280 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":523,"spinlock_wait_cycles":14592,"thread_start_us":703,"threads_started":1}
I20260926 03:49:00.881271 21119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:49:00.867605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59492 (local address 127.0.194.194:44431)
0926 03:49:00.868116 (+ 511us) server_negotiation.cc:206] Beginning negotiation
0926 03:49:00.868126 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 03:49:00.868160 (+ 34us) server_negotiation.cc:407] Connection header received
0926 03:49:00.868254 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:49:00.868269 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:49:00.868476 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:49:00.868661 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:49:00.869289 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:49:00.871325 (+ 2036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:49:00.875147 (+ 3822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:49:00.877585 (+ 2438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:49:00.877779 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:49:00.878151 (+ 372us) server_negotiation.cc:1035] Waiting for connection context
0926 03:49:00.880492 (+ 2341us) server_negotiation.cc:299] Negotiation successful
0926 03:49:00.880761 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":366,"thread_start_us":1459,"threads_started":1}
I20260926 03:49:00.881816 21118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:49:00.866584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.193:36963 (local address 127.0.0.1:36452)
0926 03:49:00.867818 (+ 1234us) negotiation.cc:107] Waiting for socket to connect
0926 03:49:00.867863 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 03:49:00.868050 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:49:00.869052 (+ 1002us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:49:00.869056 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:49:00.869081 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:49:00.870987 (+ 1906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:49:00.871004 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:49:00.874998 (+ 3994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:49:00.875007 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:49:00.879435 (+ 4428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:49:00.879447 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:49:00.881003 (+ 1556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:49:00.881009 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:49:00.881185 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:49:00.881216 (+ 31us) client_negotiation.cc:769] Sending connection context
0926 03:49:00.881304 (+ 88us) client_negotiation.cc:240] Negotiation successful
0926 03:49:00.881444 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":960,"spinlock_wait_cycles":17408,"thread_start_us":1794,"threads_started":1}
I20260926 03:49:00.881877 21116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:49:00.866617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36452 (local address 127.0.194.193:36963)
0926 03:49:00.867428 (+ 811us) server_negotiation.cc:206] Beginning negotiation
0926 03:49:00.867438 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 03:49:00.868235 (+ 797us) server_negotiation.cc:407] Connection header received
0926 03:49:00.868347 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:49:00.868354 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:49:00.868591 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:49:00.868763 (+ 172us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:49:00.871244 (+ 2481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:49:00.874774 (+ 3530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:49:00.879639 (+ 4865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:49:00.880807 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:49:00.880970 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:49:00.881265 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0926 03:49:00.881355 (+ 90us) server_negotiation.cc:299] Negotiation successful
0926 03:49:00.881508 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":619,"thread_start_us":481,"threads_started":1}
I20260926 03:49:00.885818 20778 tablet_service.cc:1467] Processing CreateTablet for tablet ac6e98036f26419296f1097f00a86673 (DEFAULT_TABLE table=test-table [id=d7a9f60b845a4eb5aff81e032da667d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:49:00.886117 20909 tablet_service.cc:1467] Processing CreateTablet for tablet ac6e98036f26419296f1097f00a86673 (DEFAULT_TABLE table=test-table [id=d7a9f60b845a4eb5aff81e032da667d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:49:00.887225 20778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac6e98036f26419296f1097f00a86673. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:49:00.887513 20909 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac6e98036f26419296f1097f00a86673. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:49:00.892515 21120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:49:00.868718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.195:45721 (local address 127.0.0.1:47356)
0926 03:49:00.878772 (+ 10054us) negotiation.cc:107] Waiting for socket to connect
0926 03:49:00.878831 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0926 03:49:00.878953 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:49:00.885251 (+ 6298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:49:00.885256 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:49:00.885290 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:49:00.885623 (+ 333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:49:00.885637 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:49:00.887633 (+ 1996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:49:00.887639 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:49:00.889650 (+ 2011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:49:00.889670 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:49:00.891665 (+ 1995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:49:00.891672 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:49:00.891874 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:49:00.891907 (+ 33us) client_negotiation.cc:769] Sending connection context
0926 03:49:00.892009 (+ 102us) client_negotiation.cc:240] Negotiation successful
0926 03:49:00.892167 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9885,"thread_start_us":10319,"threads_started":1}
I20260926 03:49:00.892807 21121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:49:00.869326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47356 (local address 127.0.194.195:45721)
0926 03:49:00.884592 (+ 15266us) server_negotiation.cc:206] Beginning negotiation
0926 03:49:00.884601 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:49:00.884640 (+ 39us) server_negotiation.cc:407] Connection header received
0926 03:49:00.884738 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:49:00.884745 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:49:00.885006 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:49:00.885232 (+ 226us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:49:00.885843 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:49:00.887451 (+ 1608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:49:00.889898 (+ 2447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:49:00.891446 (+ 1548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:49:00.891588 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:49:00.891867 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0926 03:49:00.892068 (+ 201us) server_negotiation.cc:299] Negotiation successful
0926 03:49:00.892254 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15076,"thread_start_us":16126,"threads_started":1}
I20260926 03:49:00.897344 21039 tablet_service.cc:1467] Processing CreateTablet for tablet ac6e98036f26419296f1097f00a86673 (DEFAULT_TABLE table=test-table [id=d7a9f60b845a4eb5aff81e032da667d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:49:00.899135 21039 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac6e98036f26419296f1097f00a86673. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:49:00.901464 21122 tablet_bootstrap.cc:492] T ac6e98036f26419296f1097f00a86673 P 5a391051f0254261bae36e232509e650: Bootstrap starting.
I20260926 03:49:00.905270 21122 tablet_bootstrap.cc:654] T ac6e98036f26419296f1097f00a86673 P 5a391051f0254261bae36e232509e650: Neither blocks nor log segments found. Creating new log.
I20260926 03:49:00.906610 21122 log.cc:824] T ac6e98036f26419296f1097f00a86673 P 5a391051f0254261bae36e232509e650: Log is configured to *not* fsync() on all Append() calls
I20260926 03:49:00.909821 21122 tablet_bootstrap.cc:492] T ac6e98036f26419296f1097f00a86673 P 5a391051f0254261bae36e232509e650: No bootstrap required, opened a new log
I20260926 03:49:00.910109 21122 ts_tablet_manager.cc:1397] T ac6e98036f26419296f1097f00a86673 P 5a391051f0254261bae36e232509e650: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260926 03:49:00.910894 21123 tablet_bootstrap.cc:492] T ac6e98036f26419296f1097f00a86673 P ea3819de22564d2fb659e8827119e270: Bootstrap starting.
I20260926 03:49:00.914889 21123 tablet_bootstrap.cc:654] T ac6e98036f26419296f1097f00a86673 P ea3819de22564d2fb659e8827119e270: Neither blocks nor log segments found. Creating new log.
I20260926 03:49:00.916431 21123 log.cc:824] T ac6e98036f26419296f1097f00a86673 P ea3819de22564d2fb659e8827119e270: Log is configured to *not* fsync() on all Append() calls
I20260926 03:49:00.920254 21123 tablet_bootstrap.cc:492] T ac6e98036f26419296f1097f00a86673 P ea3819de22564d2fb659e8827119e270: No bootstrap required, opened a new log
I20260926 03:49:00.920568 21125 tablet_bootstrap.cc:492] T ac6e98036f26419296f1097f00a86673 P fe64bd86a31547909778a43df40ade18: Bootstrap starting.
I20260926 03:49:00.920614 21123 ts_tablet_manager.cc:1397] T ac6e98036f26419296f1097f00a86673 P ea3819de22564d2fb659e8827119e270: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.006s
I20260926 03:49:00.921015 21122 raft_consensus.cc:348] T ac6e98036f26419296f1097f00a86673 P 5a391051f0254261bae36e232509e650 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a391051f0254261bae36e232509e650" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 36963 } } peers { permanent_uuid: "ea3819de22564d2fb659e8827119e270" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 44431 } } peers { permanent_uuid: "fe64bd86a31547909778a43df40ade18" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 45721 } }
I20260926 03:49:00.921557 21122 raft_consensus.cc:374] T ac6e98036f26419296f1097f00a86673 P 5a391051f0254261bae36e232509e650 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:49:00.921717 21122 raft_consensus.cc:729] T ac6e98036f26419296f1097f00a86673 P 5a391051f0254261bae36e232509e650 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a391051f0254261bae36e232509e650, State: Initialized, Role: FOLLOWER
I20260926 03:49:00.922447 21122 consensus_queue.cc:260] T ac6e98036f26419296f1097f00a86673 P 5a391051f0254261bae36e232509e650 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a391051f0254261bae36e232509e650" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 36963 } } peers { permanent_uuid: "ea3819de22564d2fb659e8827119e270" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 44431 } } peers { permanent_uuid: "fe64bd86a31547909778a43df40ade18" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 45721 } }
I20260926 03:49:00.924261 21125 tablet_bootstrap.cc:654] T ac6e98036f26419296f1097f00a86673 P fe64bd86a31547909778a43df40ade18: Neither blocks nor log segments found. Creating new log.
I20260926 03:49:00.924425 21122 ts_tablet_manager.cc:1428] T ac6e98036f26419296f1097f00a86673 P 5a391051f0254261bae36e232509e650: Time spent starting tablet: real 0.014s user 0.015s sys 0.000s
I20260926 03:49:00.925613 21125 log.cc:824] T ac6e98036f26419296f1097f00a86673 P fe64bd86a31547909778a43df40ade18: Log is configured to *not* fsync() on all Append() calls
I20260926 03:49:00.929596 21125 tablet_bootstrap.cc:492] T ac6e98036f26419296f1097f00a86673 P fe64bd86a31547909778a43df40ade18: No bootstrap required, opened a new log
I20260926 03:49:00.929926 21125 ts_tablet_manager.cc:1397] T ac6e98036f26419296f1097f00a86673 P fe64bd86a31547909778a43df40ade18: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.000s
I20260926 03:49:00.932487 21123 raft_consensus.cc:348] T ac6e98036f26419296f1097f00a86673 P ea3819de22564d2fb659e8827119e270 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a391051f0254261bae36e232509e650" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 36963 } } peers { permanent_uuid: "ea3819de22564d2fb659e8827119e270" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 44431 } } peers { permanent_uuid: "fe64bd86a31547909778a43df40ade18" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 45721 } }
I20260926 03:49:00.932878 21123 raft_consensus.cc:374] T ac6e98036f26419296f1097f00a86673 P ea3819de22564d2fb659e8827119e270 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:49:00.932991 21123 raft_consensus.cc:729] T ac6e98036f26419296f1097f00a86673 P ea3819de22564d2fb659e8827119e270 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea3819de22564d2fb659e8827119e270, State: Initialized, Role: FOLLOWER
I20260926 03:49:00.933665 21123 consensus_queue.cc:260] T ac6e98036f26419296f1097f00a86673 P ea3819de22564d2fb659e8827119e270 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a391051f0254261bae36e232509e650" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 36963 } } peers { permanent_uuid: "ea3819de22564d2fb659e8827119e270" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 44431 } } peers { permanent_uuid: "fe64bd86a31547909778a43df40ade18" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 45721 } }
I20260926 03:49:00.934909 20974 heartbeater.cc:503] Master 127.0.194.254:38491 was elected leader, sending a full tablet report...
I20260926 03:49:00.935487 21123 ts_tablet_manager.cc:1428] T ac6e98036f26419296f1097f00a86673 P ea3819de22564d2fb659e8827119e270: Time spent starting tablet: real 0.015s user 0.010s sys 0.002s
I20260926 03:49:00.939037 21125 raft_consensus.cc:348] T ac6e98036f26419296f1097f00a86673 P fe64bd86a31547909778a43df40ade18 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a391051f0254261bae36e232509e650" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 36963 } } peers { permanent_uuid: "ea3819de22564d2fb659e8827119e270" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 44431 } } peers { permanent_uuid: "fe64bd86a31547909778a43df40ade18" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 45721 } }
I20260926 03:49:00.939616 21125 raft_consensus.cc:374] T ac6e98036f26419296f1097f00a86673 P fe64bd86a31547909778a43df40ade18 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:49:00.939738 21125 raft_consensus.cc:729] T ac6e98036f26419296f1097f00a86673 P fe64bd86a31547909778a43df40ade18 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe64bd86a31547909778a43df40ade18, State: Initialized, Role: FOLLOWER
I20260926 03:49:00.940501 21125 consensus_queue.cc:260] T ac6e98036f26419296f1097f00a86673 P fe64bd86a31547909778a43df40ade18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a391051f0254261bae36e232509e650" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 36963 } } peers { permanent_uuid: "ea3819de22564d2fb659e8827119e270" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 44431 } } peers { permanent_uuid: "fe64bd86a31547909778a43df40ade18" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 45721 } }
I20260926 03:49:00.941627 21104 heartbeater.cc:503] Master 127.0.194.254:38491 was elected leader, sending a full tablet report...
I20260926 03:49:00.942394 21125 ts_tablet_manager.cc:1428] T ac6e98036f26419296f1097f00a86673 P fe64bd86a31547909778a43df40ade18: Time spent starting tablet: real 0.012s user 0.009s sys 0.004s
I20260926 03:49:01.023186 21127 raft_consensus.cc:482] T ac6e98036f26419296f1097f00a86673 P 5a391051f0254261bae36e232509e650 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 03:49:01.023584 21127 raft_consensus.cc:504] T ac6e98036f26419296f1097f00a86673 P 5a391051f0254261bae36e232509e650 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a391051f0254261bae36e232509e650" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 36963 } } peers { permanent_uuid: "ea3819de22564d2fb659e8827119e270" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 44431 } } peers { permanent_uuid: "fe64bd86a31547909778a43df40ade18" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 45721 } }
I20260926 03:49:01.025576 21127 leader_election.cc:288] T ac6e98036f26419296f1097f00a86673 P 5a391051f0254261bae36e232509e650 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea3819de22564d2fb659e8827119e270 (127.0.194.194:44431), fe64bd86a31547909778a43df40ade18 (127.0.194.195:45721)
I20260926 03:49:01.034039 21119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:49:01.026112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.193:48615 (local address 127.0.194.194:44431)
0926 03:49:01.026314 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0926 03:49:01.026320 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:49:01.027166 (+ 846us) server_negotiation.cc:407] Connection header received
0926 03:49:01.027359 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:49:01.027366 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:49:01.027470 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:49:01.027575 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:49:01.028532 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:49:01.029632 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:49:01.032205 (+ 2573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:49:01.032994 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:49:01.033128 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:49:01.033180 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0926 03:49:01.033597 (+ 417us) server_negotiation.cc:299] Negotiation successful
0926 03:49:01.033754 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 03:49:01.034143 21132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:49:01.026072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.194:44431 (local address 127.0.194.193:48615)
0926 03:49:01.026967 (+ 895us) negotiation.cc:107] Waiting for socket to connect
0926 03:49:01.027002 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 03:49:01.027179 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:49:01.027609 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:49:01.027614 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:49:01.027668 (+ 54us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:49:01.028320 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:49:01.028340 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:49:01.029791 (+ 1451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:49:01.029797 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:49:01.031954 (+ 2157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:49:01.031973 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:49:01.033190 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:49:01.033197 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:49:01.033376 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:49:01.033447 (+ 71us) client_negotiation.cc:769] Sending connection context
0926 03:49:01.033544 (+ 97us) client_negotiation.cc:240] Negotiation successful
0926 03:49:01.033672 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":748,"mutex_wait_us":208,"spinlock_wait_cycles":28928,"thread_start_us":435,"threads_started":1}
I20260926 03:49:01.034130 21121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:49:01.026131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.194.193:35541 (local address 127.0.194.195:45721)
0926 03:49:01.026314 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0926 03:49:01.026320 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:49:01.027166 (+ 846us) server_negotiation.cc:407] Connection header received
0926 03:49:01.027371 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:49:01.027376 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:49:01.027460 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:49:01.027575 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:49:01.028532 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:49:01.029632 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:49:01.032259 (+ 2627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:49:01.033196 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:49:01.033363 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:49:01.033438 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 03:49:01.033728 (+ 290us) server_negotiation.cc:299] Negotiation successful
0926 03:49:01.033864 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260926 03:49:01.034228 21131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:49:01.026072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.195:45721 (local address 127.0.194.193:35541)
0926 03:49:01.026998 (+ 926us) negotiation.cc:107] Waiting for socket to connect
0926 03:49:01.027012 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 03:49:01.027234 (+ 222us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:49:01.027609 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:49:01.027614 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:49:01.027643 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:49:01.028349 (+ 706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:49:01.028358 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:49:01.029843 (+ 1485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:49:01.029851 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:49:01.031972 (+ 2121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:49:01.032010 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:49:01.033382 (+ 1372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:49:01.033388 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:49:01.033555 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:49:01.033587 (+ 32us) client_negotiation.cc:769] Sending connection context
0926 03:49:01.033677 (+ 90us) client_negotiation.cc:240] Negotiation successful
0926 03:49:01.033799 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":826,"mutex_wait_us":225,"spinlock_wait_cycles":25856,"thread_start_us":462,"threads_started":1}
I20260926 03:49:01.035348 21059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac6e98036f26419296f1097f00a86673" candidate_uuid: "5a391051f0254261bae36e232509e650" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe64bd86a31547909778a43df40ade18" is_pre_election: true
I20260926 03:49:01.035457 20929 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac6e98036f26419296f1097f00a86673" candidate_uuid: "5a391051f0254261bae36e232509e650" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea3819de22564d2fb659e8827119e270" is_pre_election: true
I20260926 03:49:01.035907 21059 raft_consensus.cc:2454] T ac6e98036f26419296f1097f00a86673 P fe64bd86a31547909778a43df40ade18 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a391051f0254261bae36e232509e650 in term 0.
I20260926 03:49:01.035923 20929 raft_consensus.cc:2454] T ac6e98036f26419296f1097f00a86673 P ea3819de22564d2fb659e8827119e270 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a391051f0254261bae36e232509e650 in term 0.
I20260926 03:49:01.036633 20733 leader_election.cc:302] T ac6e98036f26419296f1097f00a86673 P 5a391051f0254261bae36e232509e650 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a391051f0254261bae36e232509e650, ea3819de22564d2fb659e8827119e270; no voters:
I20260926 03:49:01.037071 21127 raft_consensus.cc:2781] T ac6e98036f26419296f1097f00a86673 P 5a391051f0254261bae36e232509e650 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 03:49:01.037240 21127 raft_consensus.cc:482] T ac6e98036f26419296f1097f00a86673 P 5a391051f0254261bae36e232509e650 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 03:49:01.037374 21127 raft_consensus.cc:3037] T ac6e98036f26419296f1097f00a86673 P 5a391051f0254261bae36e232509e650 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:49:01.039743 21127 raft_consensus.cc:504] T ac6e98036f26419296f1097f00a86673 P 5a391051f0254261bae36e232509e650 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a391051f0254261bae36e232509e650" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 36963 } } peers { permanent_uuid: "ea3819de22564d2fb659e8827119e270" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 44431 } } peers { permanent_uuid: "fe64bd86a31547909778a43df40ade18" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 45721 } }
I20260926 03:49:01.040448 21127 leader_election.cc:288] T ac6e98036f26419296f1097f00a86673 P 5a391051f0254261bae36e232509e650 [CANDIDATE]: Term 1 election: Requested vote from peers ea3819de22564d2fb659e8827119e270 (127.0.194.194:44431), fe64bd86a31547909778a43df40ade18 (127.0.194.195:45721)
I20260926 03:49:01.040956 20929 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac6e98036f26419296f1097f00a86673" candidate_uuid: "5a391051f0254261bae36e232509e650" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea3819de22564d2fb659e8827119e270"
I20260926 03:49:01.041137 21059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac6e98036f26419296f1097f00a86673" candidate_uuid: "5a391051f0254261bae36e232509e650" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe64bd86a31547909778a43df40ade18"
I20260926 03:49:01.041196 20929 raft_consensus.cc:3037] T ac6e98036f26419296f1097f00a86673 P ea3819de22564d2fb659e8827119e270 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:49:01.041365 21059 raft_consensus.cc:3037] T ac6e98036f26419296f1097f00a86673 P fe64bd86a31547909778a43df40ade18 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:49:01.044677 20929 raft_consensus.cc:2454] T ac6e98036f26419296f1097f00a86673 P ea3819de22564d2fb659e8827119e270 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a391051f0254261bae36e232509e650 in term 1.
I20260926 03:49:01.044677 21059 raft_consensus.cc:2454] T ac6e98036f26419296f1097f00a86673 P fe64bd86a31547909778a43df40ade18 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a391051f0254261bae36e232509e650 in term 1.
I20260926 03:49:01.045219 20733 leader_election.cc:302] T ac6e98036f26419296f1097f00a86673 P 5a391051f0254261bae36e232509e650 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a391051f0254261bae36e232509e650, ea3819de22564d2fb659e8827119e270; no voters:
I20260926 03:49:01.045564 21127 raft_consensus.cc:2781] T ac6e98036f26419296f1097f00a86673 P 5a391051f0254261bae36e232509e650 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:49:01.046168 21127 raft_consensus.cc:686] T ac6e98036f26419296f1097f00a86673 P 5a391051f0254261bae36e232509e650 [term 1 LEADER]: Becoming Leader. State: Replica: 5a391051f0254261bae36e232509e650, State: Running, Role: LEADER
I20260926 03:49:01.046806 21127 consensus_queue.cc:237] T ac6e98036f26419296f1097f00a86673 P 5a391051f0254261bae36e232509e650 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a391051f0254261bae36e232509e650" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 36963 } } peers { permanent_uuid: "ea3819de22564d2fb659e8827119e270" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 44431 } } peers { permanent_uuid: "fe64bd86a31547909778a43df40ade18" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 45721 } }
W20260926 03:49:01.049342 21105 tablet.cc:2369] T ac6e98036f26419296f1097f00a86673 P fe64bd86a31547909778a43df40ade18: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 03:49:01.051724 20656 catalog_manager.cc:5495] T ac6e98036f26419296f1097f00a86673 P 5a391051f0254261bae36e232509e650 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5a391051f0254261bae36e232509e650 (127.0.194.193). New cstate: current_term: 1 leader_uuid: "5a391051f0254261bae36e232509e650" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a391051f0254261bae36e232509e650" member_type: VOTER last_known_addr { host: "127.0.194.193" port: 36963 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ea3819de22564d2fb659e8827119e270" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 44431 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe64bd86a31547909778a43df40ade18" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 45721 } health_report { overall_health: UNKNOWN } } }
W20260926 03:49:01.075416 20975 tablet.cc:2369] T ac6e98036f26419296f1097f00a86673 P ea3819de22564d2fb659e8827119e270: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 03:49:01.080160 21116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:49:01.074700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36466 (local address 127.0.194.193:36963)
0926 03:49:01.074946 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0926 03:49:01.074955 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:49:01.074977 (+ 22us) server_negotiation.cc:407] Connection header received
0926 03:49:01.075267 (+ 290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:49:01.075271 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:49:01.075366 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:49:01.075528 (+ 162us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:49:01.075878 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:49:01.076572 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:49:01.077968 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:49:01.078378 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:49:01.078508 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:49:01.079086 (+ 578us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:49:01.079590 (+ 504us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:49:01.079760 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0926 03:49:01.079812 (+ 52us) server_negotiation.cc:299] Negotiation successful
0926 03:49:01.079893 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260926 03:49:01.088940 21059 raft_consensus.cc:1261] T ac6e98036f26419296f1097f00a86673 P fe64bd86a31547909778a43df40ade18 [term 1 FOLLOWER]: Refusing update from remote peer 5a391051f0254261bae36e232509e650: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:49:01.088927 20929 raft_consensus.cc:1261] T ac6e98036f26419296f1097f00a86673 P ea3819de22564d2fb659e8827119e270 [term 1 FOLLOWER]: Refusing update from remote peer 5a391051f0254261bae36e232509e650: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260926 03:49:01.089123 20844 tablet.cc:2369] T ac6e98036f26419296f1097f00a86673 P 5a391051f0254261bae36e232509e650: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 03:49:01.090049 21133 consensus_queue.cc:1035] T ac6e98036f26419296f1097f00a86673 P 5a391051f0254261bae36e232509e650 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe64bd86a31547909778a43df40ade18" member_type: VOTER last_known_addr { host: "127.0.194.195" port: 45721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:49:01.090425 21127 consensus_queue.cc:1035] T ac6e98036f26419296f1097f00a86673 P 5a391051f0254261bae36e232509e650 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea3819de22564d2fb659e8827119e270" member_type: VOTER last_known_addr { host: "127.0.194.194" port: 44431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:49:01.103569 21138 mvcc.cc:205] Tried to move back new op lower bound from 7333456040289722368 to 7333456040132005888. Current Snapshot: MvccSnapshot[applied={T|T < 7333456040289722368}]
I20260926 03:49:01.116844 20657 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47320:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:49:01.117218 20657 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47320:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:49:01.121071 20657 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8d96663531b746fa8a0c6454f1c4dc4b: Sending DeleteTablet for 3 replicas of tablet ac6e98036f26419296f1097f00a86673
I20260926 03:49:01.122371 20909 tablet_service.cc:1514] Processing DeleteTablet for tablet ac6e98036f26419296f1097f00a86673 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:49:01 UTC) from {username='slave'} at 127.0.0.1:59492
I20260926 03:49:01.122431 20778 tablet_service.cc:1514] Processing DeleteTablet for tablet ac6e98036f26419296f1097f00a86673 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:49:01 UTC) from {username='slave'} at 127.0.0.1:36452
I20260926 03:49:01.122493 21039 tablet_service.cc:1514] Processing DeleteTablet for tablet ac6e98036f26419296f1097f00a86673 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:49:01 UTC) from {username='slave'} at 127.0.0.1:47356
I20260926 03:49:01.124624 779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu pid 20716
I20260926 03:49:01.135708 21145 tablet_replica.cc:330] T ac6e98036f26419296f1097f00a86673 P ea3819de22564d2fb659e8827119e270: stopping tablet replica
I20260926 03:49:01.137295 21147 tablet_replica.cc:330] T ac6e98036f26419296f1097f00a86673 P fe64bd86a31547909778a43df40ade18: stopping tablet replica
I20260926 03:49:01.137719 21145 raft_consensus.cc:2229] T ac6e98036f26419296f1097f00a86673 P ea3819de22564d2fb659e8827119e270 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:49:01.136595 21146 tablet_replica.cc:330] T ac6e98036f26419296f1097f00a86673 P 5a391051f0254261bae36e232509e650: stopping tablet replica
I20260926 03:49:01.138335 21146 raft_consensus.cc:2229] T ac6e98036f26419296f1097f00a86673 P 5a391051f0254261bae36e232509e650 [term 1 LEADER]: Raft consensus shutting down.
I20260926 03:49:01.137776 21147 raft_consensus.cc:2229] T ac6e98036f26419296f1097f00a86673 P fe64bd86a31547909778a43df40ade18 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:49:01.139226 21147 raft_consensus.cc:2258] T ac6e98036f26419296f1097f00a86673 P fe64bd86a31547909778a43df40ade18 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:49:01.139542 21145 raft_consensus.cc:2258] T ac6e98036f26419296f1097f00a86673 P ea3819de22564d2fb659e8827119e270 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:49:01.139988 21146 raft_consensus.cc:2258] T ac6e98036f26419296f1097f00a86673 P 5a391051f0254261bae36e232509e650 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:49:01.141853 21145 ts_tablet_manager.cc:1905] T ac6e98036f26419296f1097f00a86673 P ea3819de22564d2fb659e8827119e270: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:49:01.142160 21147 ts_tablet_manager.cc:1905] T ac6e98036f26419296f1097f00a86673 P fe64bd86a31547909778a43df40ade18: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:49:01.143363 21146 ts_tablet_manager.cc:1905] T ac6e98036f26419296f1097f00a86673 P 5a391051f0254261bae36e232509e650: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:49:01.149083 21145 ts_tablet_manager.cc:1918] T ac6e98036f26419296f1097f00a86673 P ea3819de22564d2fb659e8827119e270: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:49:01.149359 21145 log.cc:1196] T ac6e98036f26419296f1097f00a86673 P ea3819de22564d2fb659e8827119e270: Deleting WAL directory at /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790394221564982-779-0/minicluster-data/ts-1/wal/wals/ac6e98036f26419296f1097f00a86673
I20260926 03:49:01.149945 21145 ts_tablet_manager.cc:1939] T ac6e98036f26419296f1097f00a86673 P ea3819de22564d2fb659e8827119e270: Deleting consensus metadata
I20260926 03:49:01.151320 21146 ts_tablet_manager.cc:1918] T ac6e98036f26419296f1097f00a86673 P 5a391051f0254261bae36e232509e650: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:49:01.151540 21146 log.cc:1196] T ac6e98036f26419296f1097f00a86673 P 5a391051f0254261bae36e232509e650: Deleting WAL directory at /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790394221564982-779-0/minicluster-data/ts-0/wal/wals/ac6e98036f26419296f1097f00a86673
I20260926 03:49:01.151995 21116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:49:01.125962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36476 (local address 127.0.194.193:36963)
0926 03:49:01.126172 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0926 03:49:01.126180 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:49:01.131042 (+ 4862us) server_negotiation.cc:407] Connection header received
0926 03:49:01.131281 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:49:01.131287 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:49:01.131410 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:49:01.131522 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:49:01.136098 (+ 4576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:49:01.137413 (+ 1315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:49:01.142401 (+ 4988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:49:01.143152 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:49:01.143271 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:49:01.149941 (+ 6670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:49:01.149959 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:49:01.149983 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:49:01.150857 (+ 874us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:49:01.150949 (+ 92us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:49:01.150980 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:49:01.151093 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0926 03:49:01.151445 (+ 352us) server_negotiation.cc:299] Negotiation successful
0926 03:49:01.151579 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 03:49:01.152163 21146 ts_tablet_manager.cc:1939] T ac6e98036f26419296f1097f00a86673 P 5a391051f0254261bae36e232509e650: Deleting consensus metadata
I20260926 03:49:01.152464 20643 catalog_manager.cc:4841] TS ea3819de22564d2fb659e8827119e270 (127.0.194.194:44431): tablet ac6e98036f26419296f1097f00a86673 (table test-table [id=d7a9f60b845a4eb5aff81e032da667d8]) successfully deleted
I20260926 03:49:01.154021 20644 catalog_manager.cc:4841] TS 5a391051f0254261bae36e232509e650 (127.0.194.193:36963): tablet ac6e98036f26419296f1097f00a86673 (table test-table [id=d7a9f60b845a4eb5aff81e032da667d8]) successfully deleted
I20260926 03:49:01.154090 20838 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:49:01.155514 21147 ts_tablet_manager.cc:1918] T ac6e98036f26419296f1097f00a86673 P fe64bd86a31547909778a43df40ade18: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:49:01.155682 21147 log.cc:1196] T ac6e98036f26419296f1097f00a86673 P fe64bd86a31547909778a43df40ade18: Deleting WAL directory at /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790394221564982-779-0/minicluster-data/ts-2/wal/wals/ac6e98036f26419296f1097f00a86673
I20260926 03:49:01.156332 21147 ts_tablet_manager.cc:1939] T ac6e98036f26419296f1097f00a86673 P fe64bd86a31547909778a43df40ade18: Deleting consensus metadata
I20260926 03:49:01.158293 20644 catalog_manager.cc:4841] TS fe64bd86a31547909778a43df40ade18 (127.0.194.195:45721): tablet ac6e98036f26419296f1097f00a86673 (table test-table [id=d7a9f60b845a4eb5aff81e032da667d8]) successfully deleted
I20260926 03:49:01.253522 779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu with pid 20716
I20260926 03:49:01.269771 779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu pid 20847
I20260926 03:49:01.277005 21119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:49:01.270641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59496 (local address 127.0.194.194:44431)
0926 03:49:01.271100 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0926 03:49:01.271104 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:49:01.271119 (+ 15us) server_negotiation.cc:407] Connection header received
0926 03:49:01.271169 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:49:01.271172 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:49:01.271259 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:49:01.271372 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:49:01.271795 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:49:01.272551 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:49:01.273796 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:49:01.274208 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:49:01.274339 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:49:01.275629 (+ 1290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:49:01.275638 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:49:01.275654 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:49:01.276255 (+ 601us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:49:01.276302 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:49:01.276321 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:49:01.276453 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0926 03:49:01.276592 (+ 139us) server_negotiation.cc:299] Negotiation successful
0926 03:49:01.276696 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":354}
I20260926 03:49:01.277809 20969 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:49:01.366324 779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu with pid 20847
I20260926 03:49:01.383414 779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu pid 20977
I20260926 03:49:01.390643 21121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:49:01.384148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47366 (local address 127.0.194.195:45721)
0926 03:49:01.384333 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0926 03:49:01.384340 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:49:01.384456 (+ 116us) server_negotiation.cc:407] Connection header received
0926 03:49:01.384672 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:49:01.384677 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:49:01.384762 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:49:01.384908 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:49:01.385330 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:49:01.386123 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:49:01.387554 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:49:01.387980 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:49:01.388119 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:49:01.389281 (+ 1162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:49:01.389291 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:49:01.389306 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:49:01.389882 (+ 576us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:49:01.389932 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:49:01.389952 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:49:01.390080 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0926 03:49:01.390211 (+ 131us) server_negotiation.cc:299] Negotiation successful
0926 03:49:01.390325 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 03:49:01.391449 21099 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:49:01.480727 779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu with pid 20977
I20260926 03:49:01.497488 779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu pid 20626
I20260926 03:49:01.498668 20687 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:49:01.609295 779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu with pid 20626
2026-09-26T03:49:01Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (2874 ms)
[----------] 6 tests from EncryptionPolicyTest (22112 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790394221564982-779-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:49:01 dist-test-slave-n2th krb5kdc[21155](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:49:01 dist-test-slave-n2th krb5kdc[21155](info): set up 2 sockets
Sep 26 03:49:01 dist-test-slave-n2th krb5kdc[21155](info): commencing operation
krb5kdc: starting...
W20260926 03:49:02.719370 779 mini_kdc.cc:121] Time spent starting KDC: real 1.068s user 0.000s sys 0.033s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790394221564982-779-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790394221564982-779-0/krb5kdc/test-user.keytab.
Sep 26 03:49:02 dist-test-slave-n2th krb5kdc[21155](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394542, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:49:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:49:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790394221564982-779-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790394221564982-779-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 03:49:02.988795 779 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
/tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790394221564982-779-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790394221564982-779-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790394221564982-779-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790394221564982-779-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44767
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790394221564982-779-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44767
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790394221564982-779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790394221564982-779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790394221564982-779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:49:03.323086 21168 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:49:03.323397 21168 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:49:03.323568 21168 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:49:03.331017 21168 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:49:03.331137 21168 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:49:03.331228 21168 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:49:03.331314 21168 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:49:03.331389 21168 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:49:03.341584 21168 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790394221564982-779-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790394221564982-779-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44767
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790394221564982-779-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44767
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790394221564982-779-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790394221564982-779-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:49:03.342978 21168 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:49:03.344307 21168 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:49:03.356280 21174 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:49:03.356319 21176 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:49:03.356256 21173 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:49:03.357581 21168 server_base.cc:1034] running on GCE node
I20260926 03:49:03.358469 21168 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:49:03.360050 21168 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:49:03.361294 21168 hybrid_clock.cc:648] HybridClock initialized: now 1790394543361245 us; error 61 us; skew 500 ppm
I20260926 03:49:03.361667 21168 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:49:03 dist-test-slave-n2th krb5kdc[21155](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394543, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:49:03.371106 21168 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 03:49:03.372763 21168 webserver.cc:460] Webserver started at http://127.0.0.1:39027/ using document root <none> and password file <none>
I20260926 03:49:03.373543 21168 fs_manager.cc:359] Metadata directory not provided
I20260926 03:49:03.373709 21168 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:49:03.374048 21168 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:49:03.377288 21168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790394221564982-779-0/minicluster-data/master-0/data/instance:
uuid: "691131186acc4f3fb48f660eaaec76ef"
format_stamp: "Formatted at 2026-09-26 03:49:03 on dist-test-slave-n2th"
server_key: "74b1a84dfa7387376b9f035d4c4886c7"
server_key_iv: "80cb8e2e56a43420530cdb45da46c5ca"
server_key_version: "encryptionkey@0"
I20260926 03:49:03.378127 21168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790394221564982-779-0/minicluster-data/master-0/wal/instance:
uuid: "691131186acc4f3fb48f660eaaec76ef"
format_stamp: "Formatted at 2026-09-26 03:49:03 on dist-test-slave-n2th"
server_key: "74b1a84dfa7387376b9f035d4c4886c7"
server_key_iv: "80cb8e2e56a43420530cdb45da46c5ca"
server_key_version: "encryptionkey@0"
I20260926 03:49:03.383653 21168 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260926 03:49:03.387403 21183 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:49:03.388851 21168 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260926 03:49:03.389072 21168 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790394221564982-779-0/minicluster-data/master-0/data,/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790394221564982-779-0/minicluster-data/master-0/wal
uuid: "691131186acc4f3fb48f660eaaec76ef"
format_stamp: "Formatted at 2026-09-26 03:49:03 on dist-test-slave-n2th"
server_key: "74b1a84dfa7387376b9f035d4c4886c7"
server_key_iv: "80cb8e2e56a43420530cdb45da46c5ca"
server_key_version: "encryptionkey@0"
I20260926 03:49:03.389297 21168 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790394221564982-779-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790394221564982-779-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790394221564982-779-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:49:03.403838 21168 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:49:03.408699 21168 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:49:03.408949 21168 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:49:03.428339 21168 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44767
I20260926 03:49:03.428370 21234 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44767 every 8 connection(s)
I20260926 03:49:03.429780 21168 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790394221564982-779-0/minicluster-data/master-0/data/info.pb
I20260926 03:49:03.434054 21235 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:49:03.435303 779 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu as pid 21168
I20260926 03:49:03.435531 779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790394221564982-779-0/minicluster-data/master-0/wal/instance
I20260926 03:49:03.436381 779 external_mini_cluster.cc:1428] Setting key 5e9b8267d059ad1d41b529776662aced
I20260926 03:49:03.449013 21235 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 691131186acc4f3fb48f660eaaec76ef: Bootstrap starting.
Sep 26 03:49:03 dist-test-slave-n2th krb5kdc[21155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394542, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 03:49:03.452925 21235 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 691131186acc4f3fb48f660eaaec76ef: Neither blocks nor log segments found. Creating new log.
I20260926 03:49:03.454332 21235 log.cc:824] T 00000000000000000000000000000000 P 691131186acc4f3fb48f660eaaec76ef: Log is configured to *not* fsync() on all Append() calls
I20260926 03:49:03.457767 21235 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 691131186acc4f3fb48f660eaaec76ef: No bootstrap required, opened a new log
I20260926 03:49:03.460832 21238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:49:03.438435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48427 (local address 127.0.0.1:44767)
0926 03:49:03.439315 (+ 880us) server_negotiation.cc:206] Beginning negotiation
0926 03:49:03.439326 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 03:49:03.439369 (+ 43us) server_negotiation.cc:407] Connection header received
0926 03:49:03.441014 (+ 1645us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:49:03.441040 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:49:03.441465 (+ 425us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:49:03.441818 (+ 353us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:49:03.443636 (+ 1818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:49:03.444937 (+ 1301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:49:03.446866 (+ 1929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:49:03.447546 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:49:03.447676 (+ 130us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:49:03.452516 (+ 4840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:49:03.452529 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:49:03.452549 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:49:03.452603 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:49:03.456398 (+ 3795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:49:03.457288 (+ 890us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:49:03.457297 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:49:03.457306 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:49:03.457520 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:49:03.457963 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:49:03.457967 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:49:03.457969 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:49:03.458534 (+ 565us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:49:03.458902 (+ 368us) server_negotiation.cc:1035] Waiting for connection context
0926 03:49:03.459442 (+ 540us) server_negotiation.cc:299] Negotiation successful
0926 03:49:03.459790 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":657,"thread_start_us":455,"threads_started":1}
I20260926 03:49:03.465075 779 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 03:49:03.466257 21235 raft_consensus.cc:348] T 00000000000000000000000000000000 P 691131186acc4f3fb48f660eaaec76ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "691131186acc4f3fb48f660eaaec76ef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44767 } }
I20260926 03:49:03.466667 21235 raft_consensus.cc:374] T 00000000000000000000000000000000 P 691131186acc4f3fb48f660eaaec76ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:49:03.466815 21235 raft_consensus.cc:729] T 00000000000000000000000000000000 P 691131186acc4f3fb48f660eaaec76ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 691131186acc4f3fb48f660eaaec76ef, State: Initialized, Role: FOLLOWER
I20260926 03:49:03.467383 21235 consensus_queue.cc:260] T 00000000000000000000000000000000 P 691131186acc4f3fb48f660eaaec76ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "691131186acc4f3fb48f660eaaec76ef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44767 } }
I20260926 03:49:03.467636 21235 raft_consensus.cc:388] T 00000000000000000000000000000000 P 691131186acc4f3fb48f660eaaec76ef [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:49:03.467775 21235 raft_consensus.cc:482] T 00000000000000000000000000000000 P 691131186acc4f3fb48f660eaaec76ef [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:49:03.467972 21235 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 691131186acc4f3fb48f660eaaec76ef [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:49:03.470508 21235 raft_consensus.cc:504] T 00000000000000000000000000000000 P 691131186acc4f3fb48f660eaaec76ef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "691131186acc4f3fb48f660eaaec76ef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44767 } }
I20260926 03:49:03.471151 21235 leader_election.cc:302] T 00000000000000000000000000000000 P 691131186acc4f3fb48f660eaaec76ef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 691131186acc4f3fb48f660eaaec76ef; no voters:
I20260926 03:49:03.471809 21235 leader_election.cc:288] T 00000000000000000000000000000000 P 691131186acc4f3fb48f660eaaec76ef [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:49:03.471979 21245 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 691131186acc4f3fb48f660eaaec76ef [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:49:03.473646 21245 raft_consensus.cc:686] T 00000000000000000000000000000000 P 691131186acc4f3fb48f660eaaec76ef [term 1 LEADER]: Becoming Leader. State: Replica: 691131186acc4f3fb48f660eaaec76ef, State: Running, Role: LEADER
I20260926 03:49:03.474144 21235 sys_catalog.cc:564] T 00000000000000000000000000000000 P 691131186acc4f3fb48f660eaaec76ef [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:49:03.474293 21245 consensus_queue.cc:237] T 00000000000000000000000000000000 P 691131186acc4f3fb48f660eaaec76ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "691131186acc4f3fb48f660eaaec76ef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44767 } }
I20260926 03:49:03.483958 21247 sys_catalog.cc:455] T 00000000000000000000000000000000 P 691131186acc4f3fb48f660eaaec76ef [sys.catalog]: SysCatalogTable state changed. Reason: New leader 691131186acc4f3fb48f660eaaec76ef. Latest consensus state: current_term: 1 leader_uuid: "691131186acc4f3fb48f660eaaec76ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "691131186acc4f3fb48f660eaaec76ef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44767 } } }
I20260926 03:49:03.484369 21247 sys_catalog.cc:458] T 00000000000000000000000000000000 P 691131186acc4f3fb48f660eaaec76ef [sys.catalog]: This master's current role is: LEADER
I20260926 03:49:03.485030 21246 sys_catalog.cc:455] T 00000000000000000000000000000000 P 691131186acc4f3fb48f660eaaec76ef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "691131186acc4f3fb48f660eaaec76ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "691131186acc4f3fb48f660eaaec76ef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44767 } } }
I20260926 03:49:03.485435 21246 sys_catalog.cc:458] T 00000000000000000000000000000000 P 691131186acc4f3fb48f660eaaec76ef [sys.catalog]: This master's current role is: LEADER
I20260926 03:49:03.492313 21238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:49:03.470120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54613 (local address 127.0.0.1:44767)
0926 03:49:03.470325 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0926 03:49:03.470334 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:49:03.470849 (+ 515us) server_negotiation.cc:407] Connection header received
0926 03:49:03.471016 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:49:03.471021 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:49:03.471102 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:49:03.471221 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:49:03.473354 (+ 2133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:49:03.474602 (+ 1248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:49:03.476523 (+ 1921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:49:03.477487 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:49:03.477659 (+ 172us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:49:03.481665 (+ 4006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:49:03.481675 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:49:03.481683 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:49:03.481737 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:49:03.486994 (+ 5257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:49:03.488451 (+ 1457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:49:03.488459 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:49:03.488466 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:49:03.488653 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:49:03.489415 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:49:03.489422 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:49:03.489428 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:49:03.489989 (+ 561us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:49:03.491489 (+ 1500us) server_negotiation.cc:1035] Waiting for connection context
0926 03:49:03.491692 (+ 203us) server_negotiation.cc:299] Negotiation successful
0926 03:49:03.491877 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 03:49:03.494320 21254 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:49:03.498502 21254 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:49:03.507987 21254 catalog_manager.cc:1347] Generated new cluster ID: e0814ea3845b4f489a6e58bb9da6bcf5
I20260926 03:49:03.508124 21254 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:49:03.530169 21254 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:49:03.532099 21254 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:49:03.541464 21254 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 691131186acc4f3fb48f660eaaec76ef: Generated new TSK 0
I20260926 03:49:03.542335 21254 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 03:49:03.554370 779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu pid 21168
I20260926 03:49:03.556084 21230 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:49:03.655246 779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu with pid 21168
W20260926 03:49:03.673699 21153 connection.cc:535] client connection to 127.0.0.1:44767 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44767): Connection reset by peer (error 104)
2026-09-26T03:49:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (2042 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790394221564982-779-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:49:03 dist-test-slave-n2th krb5kdc[21267](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:49:03 dist-test-slave-n2th krb5kdc[21267](info): set up 2 sockets
Sep 26 03:49:03 dist-test-slave-n2th krb5kdc[21267](info): commencing operation
krb5kdc: starting...
W20260926 03:49:04.783164 779 mini_kdc.cc:121] Time spent starting KDC: real 1.090s user 0.002s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790394221564982-779-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790394221564982-779-0/krb5kdc/test-user.keytab.
Sep 26 03:49:04 dist-test-slave-n2th krb5kdc[21267](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394544, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:49:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:49:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790394221564982-779-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790394221564982-779-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 03:49:05.044389 779 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
/tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790394221564982-779-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790394221564982-779-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790394221564982-779-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790394221564982-779-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40753
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790394221564982-779-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40753
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790394221564982-779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790394221564982-779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790394221564982-779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:49:05.378422 21280 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:49:05.378775 21280 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:49:05.378909 21280 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:49:05.386277 21280 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:49:05.386402 21280 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:49:05.386454 21280 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:49:05.386518 21280 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:49:05.386561 21280 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:49:05.396961 21280 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790394221564982-779-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790394221564982-779-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40753
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790394221564982-779-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40753
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790394221564982-779-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790394221564982-779-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:49:05.398237 21280 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:49:05.399606 21280 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:49:05.411229 21288 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:49:05.411409 21285 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:49:05.411361 21286 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:49:05.412547 21280 server_base.cc:1034] running on GCE node
I20260926 03:49:05.413331 21280 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:49:05.414825 21280 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:49:05.416059 21280 hybrid_clock.cc:648] HybridClock initialized: now 1790394545416002 us; error 53 us; skew 500 ppm
I20260926 03:49:05.416404 21280 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:49:05 dist-test-slave-n2th krb5kdc[21267](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394545, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:49:05.419531 21280 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 03:49:05.420936 21280 webserver.cc:460] Webserver started at http://127.0.0.1:37053/ using document root <none> and password file <none>
I20260926 03:49:05.421624 21280 fs_manager.cc:359] Metadata directory not provided
I20260926 03:49:05.421768 21280 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:49:05.422072 21280 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:49:05.425096 21280 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790394221564982-779-0/minicluster-data/master-0/data/instance:
uuid: "5dd1474900914eecad1aa20b89471118"
format_stamp: "Formatted at 2026-09-26 03:49:05 on dist-test-slave-n2th"
server_key: "8c006f6785021d66713e6a2d4879f6e5"
server_key_iv: "0a3c3467fffd9d606affc242f9960e08"
server_key_version: "encryptionkey@0"
I20260926 03:49:05.425855 21280 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790394221564982-779-0/minicluster-data/master-0/wal/instance:
uuid: "5dd1474900914eecad1aa20b89471118"
format_stamp: "Formatted at 2026-09-26 03:49:05 on dist-test-slave-n2th"
server_key: "8c006f6785021d66713e6a2d4879f6e5"
server_key_iv: "0a3c3467fffd9d606affc242f9960e08"
server_key_version: "encryptionkey@0"
I20260926 03:49:05.431186 21280 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260926 03:49:05.434630 21295 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:49:05.436051 21280 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 03:49:05.436240 21280 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790394221564982-779-0/minicluster-data/master-0/data,/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790394221564982-779-0/minicluster-data/master-0/wal
uuid: "5dd1474900914eecad1aa20b89471118"
format_stamp: "Formatted at 2026-09-26 03:49:05 on dist-test-slave-n2th"
server_key: "8c006f6785021d66713e6a2d4879f6e5"
server_key_iv: "0a3c3467fffd9d606affc242f9960e08"
server_key_version: "encryptionkey@0"
I20260926 03:49:05.436455 21280 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790394221564982-779-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790394221564982-779-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790394221564982-779-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:49:05.453291 21280 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:49:05.457489 21280 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:49:05.457782 21280 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:49:05.476854 21280 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40753
I20260926 03:49:05.476914 21346 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40753 every 8 connection(s)
I20260926 03:49:05.478289 21280 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790394221564982-779-0/minicluster-data/master-0/data/info.pb
I20260926 03:49:05.479784 779 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu as pid 21280
I20260926 03:49:05.479959 779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790394221564982-779-0/minicluster-data/master-0/wal/instance
I20260926 03:49:05.480499 779 external_mini_cluster.cc:1428] Setting key a62a454daf28374c5b1440076253dccf
I20260926 03:49:05.483799 21347 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:49:05.494884 21350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:49:05.482230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57967 (local address 127.0.0.1:40753)
0926 03:49:05.483097 (+ 867us) server_negotiation.cc:206] Beginning negotiation
0926 03:49:05.483110 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 03:49:05.483154 (+ 44us) server_negotiation.cc:407] Connection header received
0926 03:49:05.485025 (+ 1871us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:49:05.485049 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:49:05.485456 (+ 407us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:49:05.485806 (+ 350us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:49:05.486530 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:49:05.487949 (+ 1419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:49:05.490014 (+ 2065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:49:05.490827 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:49:05.490964 (+ 137us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:49:05.492397 (+ 1433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:49:05.492404 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:49:05.492419 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:49:05.492971 (+ 552us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:49:05.493055 (+ 84us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:49:05.493101 (+ 46us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:49:05.493205 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0926 03:49:05.493561 (+ 356us) server_negotiation.cc:299] Negotiation successful
0926 03:49:05.493752 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":651,"thread_start_us":394,"threads_started":1}
I20260926 03:49:05.498646 21347 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5dd1474900914eecad1aa20b89471118: Bootstrap starting.
I20260926 03:49:05.499464 779 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 03:49:05.503068 21347 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5dd1474900914eecad1aa20b89471118: Neither blocks nor log segments found. Creating new log.
I20260926 03:49:05.504709 21347 log.cc:824] T 00000000000000000000000000000000 P 5dd1474900914eecad1aa20b89471118: Log is configured to *not* fsync() on all Append() calls
I20260926 03:49:05.508929 21347 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5dd1474900914eecad1aa20b89471118: No bootstrap required, opened a new log
I20260926 03:49:05.513123 21350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:49:05.505635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39653 (local address 127.0.0.1:40753)
0926 03:49:05.505814 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0926 03:49:05.505820 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:49:05.506127 (+ 307us) server_negotiation.cc:407] Connection header received
0926 03:49:05.506335 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:49:05.506340 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:49:05.506422 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:49:05.506545 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:49:05.507184 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:49:05.507953 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:49:05.510039 (+ 2086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:49:05.510517 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:49:05.510621 (+ 104us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:49:05.512161 (+ 1540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:49:05.512166 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:49:05.512170 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:49:05.512198 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:49:05.512242 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:49:05.512258 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:49:05.512355 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0926 03:49:05.512660 (+ 305us) server_negotiation.cc:299] Negotiation successful
0926 03:49:05.512759 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 03:49:05.518724 21347 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5dd1474900914eecad1aa20b89471118 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5dd1474900914eecad1aa20b89471118" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40753 } }
I20260926 03:49:05.519264 21347 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5dd1474900914eecad1aa20b89471118 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:49:05.519378 21347 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5dd1474900914eecad1aa20b89471118 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dd1474900914eecad1aa20b89471118, State: Initialized, Role: FOLLOWER
I20260926 03:49:05.520175 21347 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5dd1474900914eecad1aa20b89471118 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5dd1474900914eecad1aa20b89471118" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40753 } }
I20260926 03:49:05.520439 21347 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5dd1474900914eecad1aa20b89471118 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:49:05.520555 21347 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5dd1474900914eecad1aa20b89471118 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:49:05.520695 21347 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5dd1474900914eecad1aa20b89471118 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:49:05.523200 21347 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5dd1474900914eecad1aa20b89471118 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5dd1474900914eecad1aa20b89471118" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40753 } }
I20260926 03:49:05.523686 21347 leader_election.cc:302] T 00000000000000000000000000000000 P 5dd1474900914eecad1aa20b89471118 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5dd1474900914eecad1aa20b89471118; no voters:
I20260926 03:49:05.524255 21347 leader_election.cc:288] T 00000000000000000000000000000000 P 5dd1474900914eecad1aa20b89471118 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:49:05.524458 21357 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5dd1474900914eecad1aa20b89471118 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:49:05.525147 21357 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5dd1474900914eecad1aa20b89471118 [term 1 LEADER]: Becoming Leader. State: Replica: 5dd1474900914eecad1aa20b89471118, State: Running, Role: LEADER
I20260926 03:49:05.525770 21357 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5dd1474900914eecad1aa20b89471118 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5dd1474900914eecad1aa20b89471118" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40753 } }
I20260926 03:49:05.526826 21347 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5dd1474900914eecad1aa20b89471118 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:49:05.529027 21358 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5dd1474900914eecad1aa20b89471118 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5dd1474900914eecad1aa20b89471118" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5dd1474900914eecad1aa20b89471118" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40753 } } }
I20260926 03:49:05.529410 21358 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5dd1474900914eecad1aa20b89471118 [sys.catalog]: This master's current role is: LEADER
I20260926 03:49:05.529124 21359 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5dd1474900914eecad1aa20b89471118 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5dd1474900914eecad1aa20b89471118. Latest consensus state: current_term: 1 leader_uuid: "5dd1474900914eecad1aa20b89471118" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5dd1474900914eecad1aa20b89471118" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40753 } } }
I20260926 03:49:05.529578 21359 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5dd1474900914eecad1aa20b89471118 [sys.catalog]: This master's current role is: LEADER
I20260926 03:49:05.531934 21363 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:49:05.539053 21363 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:49:05.549993 21363 catalog_manager.cc:1347] Generated new cluster ID: 656a5a9801da4479b82bdd73a46bff88
I20260926 03:49:05.550137 21363 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:49:05.564121 21363 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:49:05.566087 21363 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:49:05.576496 21363 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5dd1474900914eecad1aa20b89471118: Generated new TSK 0
I20260926 03:49:05.577394 21363 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 03:49:05.590685 779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu pid 21280
W20260926 03:49:05.592059 21342 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:57967
W20260926 03:49:05.592911 779 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu pid 21280: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 03:49:05.593111 779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu with pid 21280
W20260926 03:49:05.611255 21265 connection.cc:535] client connection to 127.0.0.1:40753 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40753): Connection reset by peer (error 104)
2026-09-26T03:49:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1936 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790394221564982-779-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:49:05 dist-test-slave-n2th krb5kdc[21378](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:49:05 dist-test-slave-n2th krb5kdc[21378](info): set up 2 sockets
Sep 26 03:49:05 dist-test-slave-n2th krb5kdc[21378](info): commencing operation
krb5kdc: starting...
W20260926 03:49:06.735163 779 mini_kdc.cc:121] Time spent starting KDC: real 1.105s user 0.000s sys 0.032s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790394221564982-779-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790394221564982-779-0/krb5kdc/test-user.keytab.
Sep 26 03:49:06 dist-test-slave-n2th krb5kdc[21378](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394546, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:49:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:49:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790394221564982-779-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790394221564982-779-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 03:49:06.998911 779 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
/tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790394221564982-779-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790394221564982-779-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790394221564982-779-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790394221564982-779-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37053
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790394221564982-779-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37053
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790394221564982-779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790394221564982-779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790394221564982-779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:49:07.339213 21391 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:49:07.339557 21391 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:49:07.339762 21391 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:49:07.347250 21391 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:49:07.347405 21391 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:49:07.347493 21391 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:49:07.347612 21391 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:49:07.347688 21391 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:49:07.357978 21391 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790394221564982-779-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790394221564982-779-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37053
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790394221564982-779-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37053
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790394221564982-779-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790394221564982-779-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:49:07.359371 21391 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:49:07.360738 21391 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:49:07.372876 21397 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:49:07.372938 21396 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:49:07.372903 21399 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:49:07.373629 21391 server_base.cc:1034] running on GCE node
I20260926 03:49:07.374488 21391 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:49:07.376086 21391 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:49:07.377319 21391 hybrid_clock.cc:648] HybridClock initialized: now 1790394547377265 us; error 40 us; skew 500 ppm
I20260926 03:49:07.377702 21391 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:49:07 dist-test-slave-n2th krb5kdc[21378](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394547, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:49:07.380872 21391 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 03:49:07.382455 21391 webserver.cc:460] Webserver started at http://127.0.0.1:41159/ using document root <none> and password file <none>
I20260926 03:49:07.383261 21391 fs_manager.cc:359] Metadata directory not provided
I20260926 03:49:07.383414 21391 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:49:07.383736 21391 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:49:07.386904 21391 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790394221564982-779-0/minicluster-data/master-0/data/instance:
uuid: "b926199c8b1240da9bf6591a1f5d3c97"
format_stamp: "Formatted at 2026-09-26 03:49:07 on dist-test-slave-n2th"
server_key: "04f160f1bae291c2cb40da66ea9a4684"
server_key_iv: "1b3a3e2933041b73ef39b3e538d41685"
server_key_version: "encryptionkey@0"
I20260926 03:49:07.387717 21391 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790394221564982-779-0/minicluster-data/master-0/wal/instance:
uuid: "b926199c8b1240da9bf6591a1f5d3c97"
format_stamp: "Formatted at 2026-09-26 03:49:07 on dist-test-slave-n2th"
server_key: "04f160f1bae291c2cb40da66ea9a4684"
server_key_iv: "1b3a3e2933041b73ef39b3e538d41685"
server_key_version: "encryptionkey@0"
I20260926 03:49:07.393003 21391 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 03:49:07.396601 21406 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:49:07.398003 21391 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 03:49:07.398213 21391 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790394221564982-779-0/minicluster-data/master-0/data,/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790394221564982-779-0/minicluster-data/master-0/wal
uuid: "b926199c8b1240da9bf6591a1f5d3c97"
format_stamp: "Formatted at 2026-09-26 03:49:07 on dist-test-slave-n2th"
server_key: "04f160f1bae291c2cb40da66ea9a4684"
server_key_iv: "1b3a3e2933041b73ef39b3e538d41685"
server_key_version: "encryptionkey@0"
I20260926 03:49:07.398432 21391 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790394221564982-779-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790394221564982-779-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790394221564982-779-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:49:07.404556 21391 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:49:07.408850 21391 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:49:07.409193 21391 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:49:07.428674 21391 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37053
I20260926 03:49:07.428714 21457 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37053 every 8 connection(s)
I20260926 03:49:07.430181 21391 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790394221564982-779-0/minicluster-data/master-0/data/info.pb
I20260926 03:49:07.434454 779 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu as pid 21391
I20260926 03:49:07.434515 21458 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:49:07.434664 779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790394221564982-779-0/minicluster-data/master-0/wal/instance
I20260926 03:49:07.435217 779 external_mini_cluster.cc:1428] Setting key 2edb4adb90c8bbe8e16af04cc0b06cae
I20260926 03:49:07.443437 21461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:49:07.437069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53049 (local address 127.0.0.1:37053)
0926 03:49:07.437948 (+ 879us) server_negotiation.cc:206] Beginning negotiation
0926 03:49:07.437960 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 03:49:07.438014 (+ 54us) server_negotiation.cc:407] Connection header received
0926 03:49:07.439776 (+ 1762us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:49:07.439801 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:49:07.440180 (+ 379us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:49:07.440554 (+ 374us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:49:07.441679 (+ 1125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:49:07.441685 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:49:07.441703 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:49:07.441728 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:49:07.441792 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:49:07.441813 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:49:07.441999 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0926 03:49:07.442125 (+ 126us) server_negotiation.cc:299] Negotiation successful
0926 03:49:07.442356 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":635,"thread_start_us":407,"threads_started":1}
I20260926 03:49:07.447395 779 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 03:49:07.448738 21458 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b926199c8b1240da9bf6591a1f5d3c97: Bootstrap starting.
I20260926 03:49:07.453100 21458 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b926199c8b1240da9bf6591a1f5d3c97: Neither blocks nor log segments found. Creating new log.
I20260926 03:49:07.454412 21458 log.cc:824] T 00000000000000000000000000000000 P b926199c8b1240da9bf6591a1f5d3c97: Log is configured to *not* fsync() on all Append() calls
I20260926 03:49:07.455469 21461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:49:07.452798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50879 (local address 127.0.0.1:37053)
0926 03:49:07.453025 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0926 03:49:07.453031 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:49:07.453395 (+ 364us) server_negotiation.cc:407] Connection header received
0926 03:49:07.453559 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:49:07.453566 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:49:07.453650 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:49:07.453768 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:49:07.454701 (+ 933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:49:07.454707 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:49:07.454712 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:49:07.454753 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:49:07.454801 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:49:07.454823 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:49:07.454965 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0926 03:49:07.455088 (+ 123us) server_negotiation.cc:299] Negotiation successful
0926 03:49:07.455158 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 03:49:07.457657 21458 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b926199c8b1240da9bf6591a1f5d3c97: No bootstrap required, opened a new log
I20260926 03:49:07.465540 21458 raft_consensus.cc:348] T 00000000000000000000000000000000 P b926199c8b1240da9bf6591a1f5d3c97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b926199c8b1240da9bf6591a1f5d3c97" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37053 } }
I20260926 03:49:07.466015 21458 raft_consensus.cc:374] T 00000000000000000000000000000000 P b926199c8b1240da9bf6591a1f5d3c97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:49:07.466125 21458 raft_consensus.cc:729] T 00000000000000000000000000000000 P b926199c8b1240da9bf6591a1f5d3c97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b926199c8b1240da9bf6591a1f5d3c97, State: Initialized, Role: FOLLOWER
I20260926 03:49:07.466800 21458 consensus_queue.cc:260] T 00000000000000000000000000000000 P b926199c8b1240da9bf6591a1f5d3c97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b926199c8b1240da9bf6591a1f5d3c97" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37053 } }
I20260926 03:49:07.467067 21458 raft_consensus.cc:388] T 00000000000000000000000000000000 P b926199c8b1240da9bf6591a1f5d3c97 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:49:07.467231 21458 raft_consensus.cc:482] T 00000000000000000000000000000000 P b926199c8b1240da9bf6591a1f5d3c97 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:49:07.467393 21458 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b926199c8b1240da9bf6591a1f5d3c97 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:49:07.469934 21458 raft_consensus.cc:504] T 00000000000000000000000000000000 P b926199c8b1240da9bf6591a1f5d3c97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b926199c8b1240da9bf6591a1f5d3c97" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37053 } }
I20260926 03:49:07.470467 21458 leader_election.cc:302] T 00000000000000000000000000000000 P b926199c8b1240da9bf6591a1f5d3c97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b926199c8b1240da9bf6591a1f5d3c97; no voters:
I20260926 03:49:07.471138 21458 leader_election.cc:288] T 00000000000000000000000000000000 P b926199c8b1240da9bf6591a1f5d3c97 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:49:07.471292 21468 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b926199c8b1240da9bf6591a1f5d3c97 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:49:07.471976 21468 raft_consensus.cc:686] T 00000000000000000000000000000000 P b926199c8b1240da9bf6591a1f5d3c97 [term 1 LEADER]: Becoming Leader. State: Replica: b926199c8b1240da9bf6591a1f5d3c97, State: Running, Role: LEADER
I20260926 03:49:07.472579 21468 consensus_queue.cc:237] T 00000000000000000000000000000000 P b926199c8b1240da9bf6591a1f5d3c97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b926199c8b1240da9bf6591a1f5d3c97" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37053 } }
I20260926 03:49:07.473407 21458 sys_catalog.cc:564] T 00000000000000000000000000000000 P b926199c8b1240da9bf6591a1f5d3c97 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:49:07.477032 21470 sys_catalog.cc:455] T 00000000000000000000000000000000 P b926199c8b1240da9bf6591a1f5d3c97 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b926199c8b1240da9bf6591a1f5d3c97. Latest consensus state: current_term: 1 leader_uuid: "b926199c8b1240da9bf6591a1f5d3c97" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b926199c8b1240da9bf6591a1f5d3c97" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37053 } } }
I20260926 03:49:07.477427 21470 sys_catalog.cc:458] T 00000000000000000000000000000000 P b926199c8b1240da9bf6591a1f5d3c97 [sys.catalog]: This master's current role is: LEADER
I20260926 03:49:07.478365 21475 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:49:07.478202 21469 sys_catalog.cc:455] T 00000000000000000000000000000000 P b926199c8b1240da9bf6591a1f5d3c97 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b926199c8b1240da9bf6591a1f5d3c97" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b926199c8b1240da9bf6591a1f5d3c97" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37053 } } }
I20260926 03:49:07.479224 21469 sys_catalog.cc:458] T 00000000000000000000000000000000 P b926199c8b1240da9bf6591a1f5d3c97 [sys.catalog]: This master's current role is: LEADER
I20260926 03:49:07.486296 21475 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:49:07.495402 21475 catalog_manager.cc:1347] Generated new cluster ID: 54ad8e0bf67d4e4ba08b2b37b05a5a32
I20260926 03:49:07.495570 21475 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:49:07.512565 21475 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:49:07.513151 21475 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:49:07.528091 21475 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b926199c8b1240da9bf6591a1f5d3c97: Generated new TSK 0
I20260926 03:49:07.529232 21475 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 03:49:07.543924 779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu pid 21391
W20260926 03:49:07.545004 21453 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:53049
W20260926 03:49:07.545725 779 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu pid 21391: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 03:49:07.545881 779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu with pid 21391
2026-09-26T03:49:07Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1951 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790394221564982-779-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:49:07 dist-test-slave-n2th krb5kdc[21489](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:49:07 dist-test-slave-n2th krb5kdc[21489](info): set up 2 sockets
Sep 26 03:49:07 dist-test-slave-n2th krb5kdc[21489](info): commencing operation
krb5kdc: starting...
W20260926 03:49:08.659197 779 mini_kdc.cc:121] Time spent starting KDC: real 1.078s user 0.007s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790394221564982-779-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790394221564982-779-0/krb5kdc/test-user.keytab.
Sep 26 03:49:08 dist-test-slave-n2th krb5kdc[21489](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394548, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:49:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:49:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790394221564982-779-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790394221564982-779-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 03:49:08.921972 779 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
/tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790394221564982-779-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790394221564982-779-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790394221564982-779-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790394221564982-779-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42959
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790394221564982-779-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42959
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790394221564982-779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790394221564982-779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790394221564982-779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:49:09.261320 21502 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:49:09.261638 21502 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:49:09.261819 21502 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:49:09.269225 21502 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:49:09.269368 21502 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:49:09.269454 21502 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:49:09.269577 21502 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:49:09.269639 21502 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:49:09.280043 21502 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790394221564982-779-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790394221564982-779-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42959
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790394221564982-779-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42959
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790394221564982-779-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790394221564982-779-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:49:09.281503 21502 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:49:09.282927 21502 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:49:09.295827 21510 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:49:09.295897 21507 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:49:09.295919 21508 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:49:09.297806 21502 server_base.cc:1034] running on GCE node
I20260926 03:49:09.298897 21502 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:49:09.300546 21502 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:49:09.301805 21502 hybrid_clock.cc:648] HybridClock initialized: now 1790394549301767 us; error 52 us; skew 500 ppm
I20260926 03:49:09.302218 21502 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:49:09 dist-test-slave-n2th krb5kdc[21489](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394549, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:49:09.310313 21502 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 03:49:09.312178 21502 webserver.cc:460] Webserver started at http://127.0.0.1:33085/ using document root <none> and password file <none>
I20260926 03:49:09.312964 21502 fs_manager.cc:359] Metadata directory not provided
I20260926 03:49:09.313126 21502 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:49:09.313465 21502 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:49:09.316851 21502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790394221564982-779-0/minicluster-data/master-0/data/instance:
uuid: "6281f79491994c29b7dbd884ff22acb0"
format_stamp: "Formatted at 2026-09-26 03:49:09 on dist-test-slave-n2th"
server_key: "fb167c284e7a9c3078b15431a5c4c19d"
server_key_iv: "d40872e741ef5ff6bfc5f57ed969b81d"
server_key_version: "encryptionkey@0"
I20260926 03:49:09.317737 21502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790394221564982-779-0/minicluster-data/master-0/wal/instance:
uuid: "6281f79491994c29b7dbd884ff22acb0"
format_stamp: "Formatted at 2026-09-26 03:49:09 on dist-test-slave-n2th"
server_key: "fb167c284e7a9c3078b15431a5c4c19d"
server_key_iv: "d40872e741ef5ff6bfc5f57ed969b81d"
server_key_version: "encryptionkey@0"
I20260926 03:49:09.323985 21502 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260926 03:49:09.328245 21517 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:49:09.330016 21502 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260926 03:49:09.330260 21502 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790394221564982-779-0/minicluster-data/master-0/data,/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790394221564982-779-0/minicluster-data/master-0/wal
uuid: "6281f79491994c29b7dbd884ff22acb0"
format_stamp: "Formatted at 2026-09-26 03:49:09 on dist-test-slave-n2th"
server_key: "fb167c284e7a9c3078b15431a5c4c19d"
server_key_iv: "d40872e741ef5ff6bfc5f57ed969b81d"
server_key_version: "encryptionkey@0"
I20260926 03:49:09.330526 21502 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790394221564982-779-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790394221564982-779-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790394221564982-779-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:49:09.347790 21502 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:49:09.352665 21502 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:49:09.352994 21502 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:49:09.371480 21502 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42959
I20260926 03:49:09.371500 21568 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42959 every 8 connection(s)
I20260926 03:49:09.372915 21502 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790394221564982-779-0/minicluster-data/master-0/data/info.pb
I20260926 03:49:09.376803 21569 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:49:09.378067 779 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu as pid 21502
I20260926 03:49:09.378245 779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790394221564982-779-0/minicluster-data/master-0/wal/instance
I20260926 03:49:09.378679 779 external_mini_cluster.cc:1428] Setting key d13c56026450b61a529b7e1b8feeebb7
I20260926 03:49:09.389633 21569 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6281f79491994c29b7dbd884ff22acb0: Bootstrap starting.
Sep 26 03:49:09 dist-test-slave-n2th krb5kdc[21489](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394548, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 03:49:09.393182 21569 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6281f79491994c29b7dbd884ff22acb0: Neither blocks nor log segments found. Creating new log.
I20260926 03:49:09.394279 21569 log.cc:824] T 00000000000000000000000000000000 P 6281f79491994c29b7dbd884ff22acb0: Log is configured to *not* fsync() on all Append() calls
I20260926 03:49:09.397411 21569 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6281f79491994c29b7dbd884ff22acb0: No bootstrap required, opened a new log
I20260926 03:49:09.401726 21572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:49:09.380617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51067 (local address 127.0.0.1:42959)
0926 03:49:09.381419 (+ 802us) server_negotiation.cc:206] Beginning negotiation
0926 03:49:09.381431 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 03:49:09.381480 (+ 49us) server_negotiation.cc:407] Connection header received
0926 03:49:09.383272 (+ 1792us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:49:09.383297 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:49:09.383759 (+ 462us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:49:09.384176 (+ 417us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:49:09.385686 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:49:09.387121 (+ 1435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:49:09.388455 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:49:09.389152 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:49:09.389358 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:49:09.393786 (+ 4428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:49:09.393795 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:49:09.393811 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:49:09.393852 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:49:09.397089 (+ 3237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:49:09.397921 (+ 832us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:49:09.397927 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:49:09.397934 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:49:09.398118 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:49:09.398685 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:49:09.398691 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:49:09.398694 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:49:09.399340 (+ 646us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:49:09.399704 (+ 364us) server_negotiation.cc:1035] Waiting for connection context
0926 03:49:09.400211 (+ 507us) server_negotiation.cc:299] Negotiation successful
0926 03:49:09.400675 (+ 464us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":557,"thread_start_us":339,"threads_started":1}
I20260926 03:49:09.405980 779 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 03:49:09.406905 21569 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6281f79491994c29b7dbd884ff22acb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6281f79491994c29b7dbd884ff22acb0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42959 } }
I20260926 03:49:09.407310 21569 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6281f79491994c29b7dbd884ff22acb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:49:09.407418 21569 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6281f79491994c29b7dbd884ff22acb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6281f79491994c29b7dbd884ff22acb0, State: Initialized, Role: FOLLOWER
I20260926 03:49:09.407981 21569 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6281f79491994c29b7dbd884ff22acb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6281f79491994c29b7dbd884ff22acb0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42959 } }
I20260926 03:49:09.408229 21569 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6281f79491994c29b7dbd884ff22acb0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:49:09.408372 21569 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6281f79491994c29b7dbd884ff22acb0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:49:09.408540 21569 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6281f79491994c29b7dbd884ff22acb0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:49:09.411139 21569 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6281f79491994c29b7dbd884ff22acb0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6281f79491994c29b7dbd884ff22acb0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42959 } }
I20260926 03:49:09.411760 21569 leader_election.cc:302] T 00000000000000000000000000000000 P 6281f79491994c29b7dbd884ff22acb0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6281f79491994c29b7dbd884ff22acb0; no voters:
I20260926 03:49:09.412395 21569 leader_election.cc:288] T 00000000000000000000000000000000 P 6281f79491994c29b7dbd884ff22acb0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:49:09.412604 21579 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6281f79491994c29b7dbd884ff22acb0 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:49:09.414417 21579 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6281f79491994c29b7dbd884ff22acb0 [term 1 LEADER]: Becoming Leader. State: Replica: 6281f79491994c29b7dbd884ff22acb0, State: Running, Role: LEADER
I20260926 03:49:09.414516 21569 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6281f79491994c29b7dbd884ff22acb0 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:49:09.415115 21579 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6281f79491994c29b7dbd884ff22acb0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6281f79491994c29b7dbd884ff22acb0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42959 } }
I20260926 03:49:09.417992 21581 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6281f79491994c29b7dbd884ff22acb0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6281f79491994c29b7dbd884ff22acb0. Latest consensus state: current_term: 1 leader_uuid: "6281f79491994c29b7dbd884ff22acb0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6281f79491994c29b7dbd884ff22acb0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42959 } } }
I20260926 03:49:09.418999 21581 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6281f79491994c29b7dbd884ff22acb0 [sys.catalog]: This master's current role is: LEADER
I20260926 03:49:09.418608 21580 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6281f79491994c29b7dbd884ff22acb0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6281f79491994c29b7dbd884ff22acb0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6281f79491994c29b7dbd884ff22acb0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42959 } } }
I20260926 03:49:09.420018 21580 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6281f79491994c29b7dbd884ff22acb0 [sys.catalog]: This master's current role is: LEADER
I20260926 03:49:09.421000 21585 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:49:09.427709 21585 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:49:09.430029 21572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:49:09.410932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51441 (local address 127.0.0.1:42959)
0926 03:49:09.411136 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0926 03:49:09.411142 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:49:09.411517 (+ 375us) server_negotiation.cc:407] Connection header received
0926 03:49:09.411644 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:49:09.411649 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:49:09.411731 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:49:09.411853 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:49:09.414130 (+ 2277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:49:09.415320 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:49:09.417155 (+ 1835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:49:09.417990 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:49:09.418140 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:49:09.421710 (+ 3570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:49:09.421720 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:49:09.421727 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:49:09.421776 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:49:09.426006 (+ 4230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:49:09.427051 (+ 1045us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:49:09.427056 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:49:09.427060 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:49:09.427220 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:49:09.427729 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:49:09.427734 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:49:09.427738 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:49:09.428161 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:49:09.429094 (+ 933us) server_negotiation.cc:1035] Waiting for connection context
0926 03:49:09.429276 (+ 182us) server_negotiation.cc:299] Negotiation successful
0926 03:49:09.429484 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 03:49:09.441550 21585 catalog_manager.cc:1347] Generated new cluster ID: 8d81023a5fda43eb9f96be58d8fb1d1d
I20260926 03:49:09.441691 21585 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:49:09.454608 21585 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:49:09.455992 21585 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:49:09.463311 21585 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6281f79491994c29b7dbd884ff22acb0: Generated new TSK 0
I20260926 03:49:09.464326 21585 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 03:49:09.477514 779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu pid 21502
I20260926 03:49:09.478989 21564 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:49:09.570871 779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu with pid 21502
2026-09-26T03:49:09Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (2026 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790394221564982-779-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:49:09 dist-test-slave-n2th krb5kdc[21601](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:49:09 dist-test-slave-n2th krb5kdc[21601](info): set up 2 sockets
Sep 26 03:49:09 dist-test-slave-n2th krb5kdc[21601](info): commencing operation
krb5kdc: starting...
W20260926 03:49:10.683331 779 mini_kdc.cc:121] Time spent starting KDC: real 1.076s user 0.005s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790394221564982-779-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790394221564982-779-0/krb5kdc/test-user.keytab.
Sep 26 03:49:10 dist-test-slave-n2th krb5kdc[21601](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394550, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:49:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:49:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790394221564982-779-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790394221564982-779-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 03:49:10.945700 779 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
/tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790394221564982-779-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790394221564982-779-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790394221564982-779-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790394221564982-779-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40149
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790394221564982-779-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38211
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40149
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790394221564982-779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790394221564982-779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790394221564982-779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:49:11.281418 21614 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:49:11.281726 21614 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:49:11.281823 21614 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:49:11.289292 21614 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:49:11.289417 21614 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:49:11.289463 21614 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:49:11.289525 21614 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:49:11.289562 21614 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:49:11.299501 21614 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38211
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790394221564982-779-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790394221564982-779-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40149
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790394221564982-779-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40149
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790394221564982-779-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790394221564982-779-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:49:11.300786 21614 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:49:11.302089 21614 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:49:11.314327 21620 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:49:11.314373 21619 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:49:11.314349 21622 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:49:11.315403 21614 server_base.cc:1034] running on GCE node
I20260926 03:49:11.316190 21614 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:49:11.317611 21614 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:49:11.318840 21614 hybrid_clock.cc:648] HybridClock initialized: now 1790394551318790 us; error 55 us; skew 500 ppm
I20260926 03:49:11.319211 21614 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:49:11 dist-test-slave-n2th krb5kdc[21601](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394551, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:49:11.322382 21614 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 03:49:11.324101 21614 webserver.cc:460] Webserver started at http://127.0.0.1:39601/ using document root <none> and password file <none>
I20260926 03:49:11.324832 21614 fs_manager.cc:359] Metadata directory not provided
I20260926 03:49:11.324983 21614 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:49:11.325307 21614 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:49:11.328512 21614 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790394221564982-779-0/minicluster-data/master-0/data/instance:
uuid: "b0fa3df485714d42a03345ede8416e07"
format_stamp: "Formatted at 2026-09-26 03:49:11 on dist-test-slave-n2th"
server_key: "456e1e4bf50e347c818e3ef49217408e"
server_key_iv: "8d465d597c904f108a05c10532e983f8"
server_key_version: "encryptionkey@0"
I20260926 03:49:11.329296 21614 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790394221564982-779-0/minicluster-data/master-0/wal/instance:
uuid: "b0fa3df485714d42a03345ede8416e07"
format_stamp: "Formatted at 2026-09-26 03:49:11 on dist-test-slave-n2th"
server_key: "456e1e4bf50e347c818e3ef49217408e"
server_key_iv: "8d465d597c904f108a05c10532e983f8"
server_key_version: "encryptionkey@0"
I20260926 03:49:11.334807 21614 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260926 03:49:11.338346 21629 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:49:11.339720 21614 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 03:49:11.339900 21614 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790394221564982-779-0/minicluster-data/master-0/data,/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790394221564982-779-0/minicluster-data/master-0/wal
uuid: "b0fa3df485714d42a03345ede8416e07"
format_stamp: "Formatted at 2026-09-26 03:49:11 on dist-test-slave-n2th"
server_key: "456e1e4bf50e347c818e3ef49217408e"
server_key_iv: "8d465d597c904f108a05c10532e983f8"
server_key_version: "encryptionkey@0"
I20260926 03:49:11.340111 21614 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790394221564982-779-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790394221564982-779-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790394221564982-779-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:49:11.357046 21614 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:49:11.361188 21614 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:49:11.361492 21614 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:49:11.380537 21614 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40149
I20260926 03:49:11.380591 21680 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40149 every 8 connection(s)
I20260926 03:49:11.382051 21614 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790394221564982-779-0/minicluster-data/master-0/data/info.pb
I20260926 03:49:11.385803 21681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:49:11.391172 779 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu as pid 21614
I20260926 03:49:11.391376 779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790394221564982-779-0/minicluster-data/master-0/wal/instance
I20260926 03:49:11.391817 779 external_mini_cluster.cc:1428] Setting key 6f443461df241e56aba414deb83d6aa4
I20260926 03:49:11.396909 21681 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0fa3df485714d42a03345ede8416e07: Bootstrap starting.
I20260926 03:49:11.400743 21681 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b0fa3df485714d42a03345ede8416e07: Neither blocks nor log segments found. Creating new log.
I20260926 03:49:11.401971 21681 log.cc:824] T 00000000000000000000000000000000 P b0fa3df485714d42a03345ede8416e07: Log is configured to *not* fsync() on all Append() calls
I20260926 03:49:11.405279 21681 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0fa3df485714d42a03345ede8416e07: No bootstrap required, opened a new log
I20260926 03:49:11.405654 21684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:49:11.393718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60713 (local address 127.0.0.1:40149)
0926 03:49:11.394586 (+ 868us) server_negotiation.cc:206] Beginning negotiation
0926 03:49:11.394599 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 03:49:11.394647 (+ 48us) server_negotiation.cc:407] Connection header received
0926 03:49:11.396400 (+ 1753us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:49:11.396423 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:49:11.396854 (+ 431us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:49:11.397237 (+ 383us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:49:11.397862 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:49:11.399249 (+ 1387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:49:11.401116 (+ 1867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:49:11.401796 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:49:11.401933 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:49:11.403290 (+ 1357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:49:11.403300 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:49:11.403320 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:49:11.403905 (+ 585us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:49:11.403981 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:49:11.404007 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:49:11.404093 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0926 03:49:11.404338 (+ 245us) server_negotiation.cc:299] Negotiation successful
0926 03:49:11.404635 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":615,"thread_start_us":397,"threads_started":1}
I20260926 03:49:11.409520 779 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 03:49:11.413867 21681 raft_consensus.cc:348] T 00000000000000000000000000000000 P b0fa3df485714d42a03345ede8416e07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0fa3df485714d42a03345ede8416e07" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40149 } }
I20260926 03:49:11.414436 21681 raft_consensus.cc:374] T 00000000000000000000000000000000 P b0fa3df485714d42a03345ede8416e07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:49:11.414578 21681 raft_consensus.cc:729] T 00000000000000000000000000000000 P b0fa3df485714d42a03345ede8416e07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0fa3df485714d42a03345ede8416e07, State: Initialized, Role: FOLLOWER
I20260926 03:49:11.415278 21681 consensus_queue.cc:260] T 00000000000000000000000000000000 P b0fa3df485714d42a03345ede8416e07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0fa3df485714d42a03345ede8416e07" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40149 } }
I20260926 03:49:11.415535 21681 raft_consensus.cc:388] T 00000000000000000000000000000000 P b0fa3df485714d42a03345ede8416e07 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:49:11.415709 21681 raft_consensus.cc:482] T 00000000000000000000000000000000 P b0fa3df485714d42a03345ede8416e07 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:49:11.415905 21681 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b0fa3df485714d42a03345ede8416e07 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:49:11.418205 21681 raft_consensus.cc:504] T 00000000000000000000000000000000 P b0fa3df485714d42a03345ede8416e07 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0fa3df485714d42a03345ede8416e07" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40149 } }
I20260926 03:49:11.418798 21681 leader_election.cc:302] T 00000000000000000000000000000000 P b0fa3df485714d42a03345ede8416e07 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b0fa3df485714d42a03345ede8416e07; no voters:
I20260926 03:49:11.419480 21681 leader_election.cc:288] T 00000000000000000000000000000000 P b0fa3df485714d42a03345ede8416e07 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:49:11.419672 21691 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b0fa3df485714d42a03345ede8416e07 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:49:11.421082 21691 raft_consensus.cc:686] T 00000000000000000000000000000000 P b0fa3df485714d42a03345ede8416e07 [term 1 LEADER]: Becoming Leader. State: Replica: b0fa3df485714d42a03345ede8416e07, State: Running, Role: LEADER
I20260926 03:49:11.421367 21684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:49:11.413878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49251 (local address 127.0.0.1:40149)
0926 03:49:11.414080 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0926 03:49:11.414090 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 03:49:11.414489 (+ 399us) server_negotiation.cc:407] Connection header received
0926 03:49:11.414717 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:49:11.414723 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:49:11.414829 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:49:11.414937 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:49:11.415615 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:49:11.416508 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:49:11.417866 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:49:11.418332 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:49:11.418477 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:49:11.419961 (+ 1484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:49:11.419970 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:49:11.419981 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:49:11.420016 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:49:11.420082 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:49:11.420106 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:49:11.420223 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0926 03:49:11.420826 (+ 603us) server_negotiation.cc:299] Negotiation successful
0926 03:49:11.420922 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 03:49:11.421869 21681 sys_catalog.cc:564] T 00000000000000000000000000000000 P b0fa3df485714d42a03345ede8416e07 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:49:11.422284 21691 consensus_queue.cc:237] T 00000000000000000000000000000000 P b0fa3df485714d42a03345ede8416e07 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0fa3df485714d42a03345ede8416e07" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40149 } }
I20260926 03:49:11.425568 21692 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0fa3df485714d42a03345ede8416e07 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b0fa3df485714d42a03345ede8416e07" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0fa3df485714d42a03345ede8416e07" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40149 } } }
I20260926 03:49:11.425547 21693 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0fa3df485714d42a03345ede8416e07 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b0fa3df485714d42a03345ede8416e07. Latest consensus state: current_term: 1 leader_uuid: "b0fa3df485714d42a03345ede8416e07" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0fa3df485714d42a03345ede8416e07" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40149 } } }
I20260926 03:49:11.426002 21692 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0fa3df485714d42a03345ede8416e07 [sys.catalog]: This master's current role is: LEADER
I20260926 03:49:11.426023 21693 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0fa3df485714d42a03345ede8416e07 [sys.catalog]: This master's current role is: LEADER
I20260926 03:49:11.428334 21698 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:49:11.434336 21698 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:49:11.444617 21698 catalog_manager.cc:1347] Generated new cluster ID: 454d9b0d61fb4ae2bb62b91678eb6228
I20260926 03:49:11.444764 21698 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:49:11.478664 21698 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:49:11.480355 21698 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:49:11.493605 21698 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b0fa3df485714d42a03345ede8416e07: Generated new TSK 0
I20260926 03:49:11.494498 21698 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 03:49:11.510057 779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu pid 21614
W20260926 03:49:11.511289 21676 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:60713
W20260926 03:49:11.512034 779 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu pid 21614: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 03:49:11.512187 779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu with pid 21614
2026-09-26T03:49:11Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1940 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790394221564982-779-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:49:11 dist-test-slave-n2th krb5kdc[21712](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:49:11 dist-test-slave-n2th krb5kdc[21712](info): set up 2 sockets
Sep 26 03:49:11 dist-test-slave-n2th krb5kdc[21712](info): commencing operation
krb5kdc: starting...
W20260926 03:49:12.627252 779 mini_kdc.cc:121] Time spent starting KDC: real 1.079s user 0.002s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790394221564982-779-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790394221564982-779-0/krb5kdc/test-user.keytab.
Sep 26 03:49:12 dist-test-slave-n2th krb5kdc[21712](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394552, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:49:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:49:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790394221564982-779-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790394221564982-779-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 03:49:12.894491 779 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
/tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790394221564982-779-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790394221564982-779-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790394221564982-779-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790394221564982-779-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44875
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790394221564982-779-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44875
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790394221564982-779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790394221564982-779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790394221564982-779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:49:13.236498 21725 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:49:13.236847 21725 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:49:13.237018 21725 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:49:13.244419 21725 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:49:13.244604 21725 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:49:13.244709 21725 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:49:13.244812 21725 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:49:13.244894 21725 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:49:13.255266 21725 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790394221564982-779-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790394221564982-779-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44875
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790394221564982-779-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44875
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790394221564982-779-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790394221564982-779-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:49:13.256637 21725 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:49:13.258025 21725 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:49:13.270234 21733 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:49:13.270144 21731 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:49:13.270274 21730 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:49:13.271190 21725 server_base.cc:1034] running on GCE node
I20260926 03:49:13.272100 21725 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:49:13.273555 21725 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:49:13.274822 21725 hybrid_clock.cc:648] HybridClock initialized: now 1790394553274760 us; error 43 us; skew 500 ppm
I20260926 03:49:13.275179 21725 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:49:13 dist-test-slave-n2th krb5kdc[21712](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394553, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:49:13.278332 21725 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 03:49:13.279820 21725 webserver.cc:460] Webserver started at http://127.0.0.1:40357/ using document root <none> and password file <none>
I20260926 03:49:13.280520 21725 fs_manager.cc:359] Metadata directory not provided
I20260926 03:49:13.280666 21725 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:49:13.280975 21725 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:49:13.284056 21725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790394221564982-779-0/minicluster-data/master-0/data/instance:
uuid: "cd24e1280a20445b8514fca298376f13"
format_stamp: "Formatted at 2026-09-26 03:49:13 on dist-test-slave-n2th"
server_key: "15a3ea89b645d8dd13a88af0c4c11933"
server_key_iv: "e6daf61204cfd1d4205d4b680731f649"
server_key_version: "encryptionkey@0"
I20260926 03:49:13.284854 21725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790394221564982-779-0/minicluster-data/master-0/wal/instance:
uuid: "cd24e1280a20445b8514fca298376f13"
format_stamp: "Formatted at 2026-09-26 03:49:13 on dist-test-slave-n2th"
server_key: "15a3ea89b645d8dd13a88af0c4c11933"
server_key_iv: "e6daf61204cfd1d4205d4b680731f649"
server_key_version: "encryptionkey@0"
I20260926 03:49:13.290300 21725 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260926 03:49:13.294071 21740 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:49:13.295500 21725 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 03:49:13.295697 21725 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790394221564982-779-0/minicluster-data/master-0/data,/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790394221564982-779-0/minicluster-data/master-0/wal
uuid: "cd24e1280a20445b8514fca298376f13"
format_stamp: "Formatted at 2026-09-26 03:49:13 on dist-test-slave-n2th"
server_key: "15a3ea89b645d8dd13a88af0c4c11933"
server_key_iv: "e6daf61204cfd1d4205d4b680731f649"
server_key_version: "encryptionkey@0"
I20260926 03:49:13.295917 21725 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790394221564982-779-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790394221564982-779-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790394221564982-779-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:49:13.301900 21725 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:49:13.306564 21725 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:49:13.306965 21725 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:49:13.326244 21725 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44875
I20260926 03:49:13.326287 21791 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44875 every 8 connection(s)
I20260926 03:49:13.327762 21725 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790394221564982-779-0/minicluster-data/master-0/data/info.pb
I20260926 03:49:13.331696 21792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:49:13.332532 779 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu as pid 21725
I20260926 03:49:13.332726 779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790394221564982-779-0/minicluster-data/master-0/wal/instance
I20260926 03:49:13.333267 779 external_mini_cluster.cc:1428] Setting key 3f89c0a39c6ff2f73982a0daeeeb3319
I20260926 03:49:13.341835 21795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:49:13.335443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51795 (local address 127.0.0.1:44875)
0926 03:49:13.336389 (+ 946us) server_negotiation.cc:206] Beginning negotiation
0926 03:49:13.336400 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 03:49:13.336443 (+ 43us) server_negotiation.cc:407] Connection header received
0926 03:49:13.338380 (+ 1937us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:49:13.338404 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:49:13.338859 (+ 455us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:49:13.339220 (+ 361us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:49:13.340267 (+ 1047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:49:13.340273 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:49:13.340287 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:49:13.340301 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:49:13.340359 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:49:13.340382 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:49:13.340556 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0926 03:49:13.340654 (+ 98us) server_negotiation.cc:299] Negotiation successful
0926 03:49:13.340900 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":707,"thread_start_us":465,"threads_started":1}
I20260926 03:49:13.346135 779 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 03:49:13.346200 21792 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd24e1280a20445b8514fca298376f13: Bootstrap starting.
I20260926 03:49:13.350340 21792 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cd24e1280a20445b8514fca298376f13: Neither blocks nor log segments found. Creating new log.
I20260926 03:49:13.351778 21792 log.cc:824] T 00000000000000000000000000000000 P cd24e1280a20445b8514fca298376f13: Log is configured to *not* fsync() on all Append() calls
I20260926 03:49:13.354467 21795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:49:13.351637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58181 (local address 127.0.0.1:44875)
0926 03:49:13.351861 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0926 03:49:13.351869 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:49:13.352187 (+ 318us) server_negotiation.cc:407] Connection header received
0926 03:49:13.352449 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:49:13.352458 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:49:13.352543 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:49:13.352665 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:49:13.353670 (+ 1005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:49:13.353674 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:49:13.353677 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:49:13.353693 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:49:13.353733 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:49:13.353748 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:49:13.353859 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0926 03:49:13.354053 (+ 194us) server_negotiation.cc:299] Negotiation successful
0926 03:49:13.354144 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260926 03:49:13.355453 21792 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd24e1280a20445b8514fca298376f13: No bootstrap required, opened a new log
I20260926 03:49:13.363152 21792 raft_consensus.cc:348] T 00000000000000000000000000000000 P cd24e1280a20445b8514fca298376f13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd24e1280a20445b8514fca298376f13" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44875 } }
I20260926 03:49:13.363580 21792 raft_consensus.cc:374] T 00000000000000000000000000000000 P cd24e1280a20445b8514fca298376f13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:49:13.363709 21792 raft_consensus.cc:729] T 00000000000000000000000000000000 P cd24e1280a20445b8514fca298376f13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd24e1280a20445b8514fca298376f13, State: Initialized, Role: FOLLOWER
I20260926 03:49:13.364357 21792 consensus_queue.cc:260] T 00000000000000000000000000000000 P cd24e1280a20445b8514fca298376f13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd24e1280a20445b8514fca298376f13" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44875 } }
I20260926 03:49:13.364616 21792 raft_consensus.cc:388] T 00000000000000000000000000000000 P cd24e1280a20445b8514fca298376f13 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:49:13.364774 21792 raft_consensus.cc:482] T 00000000000000000000000000000000 P cd24e1280a20445b8514fca298376f13 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:49:13.364961 21792 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cd24e1280a20445b8514fca298376f13 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:49:13.367300 21792 raft_consensus.cc:504] T 00000000000000000000000000000000 P cd24e1280a20445b8514fca298376f13 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd24e1280a20445b8514fca298376f13" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44875 } }
I20260926 03:49:13.367766 21792 leader_election.cc:302] T 00000000000000000000000000000000 P cd24e1280a20445b8514fca298376f13 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cd24e1280a20445b8514fca298376f13; no voters:
I20260926 03:49:13.368335 21792 leader_election.cc:288] T 00000000000000000000000000000000 P cd24e1280a20445b8514fca298376f13 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:49:13.368454 21802 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cd24e1280a20445b8514fca298376f13 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:49:13.369644 21802 raft_consensus.cc:686] T 00000000000000000000000000000000 P cd24e1280a20445b8514fca298376f13 [term 1 LEADER]: Becoming Leader. State: Replica: cd24e1280a20445b8514fca298376f13, State: Running, Role: LEADER
I20260926 03:49:13.370271 21802 consensus_queue.cc:237] T 00000000000000000000000000000000 P cd24e1280a20445b8514fca298376f13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd24e1280a20445b8514fca298376f13" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44875 } }
I20260926 03:49:13.370637 21792 sys_catalog.cc:564] T 00000000000000000000000000000000 P cd24e1280a20445b8514fca298376f13 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:49:13.372825 21804 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd24e1280a20445b8514fca298376f13 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cd24e1280a20445b8514fca298376f13. Latest consensus state: current_term: 1 leader_uuid: "cd24e1280a20445b8514fca298376f13" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd24e1280a20445b8514fca298376f13" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44875 } } }
I20260926 03:49:13.373176 21804 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd24e1280a20445b8514fca298376f13 [sys.catalog]: This master's current role is: LEADER
I20260926 03:49:13.373449 21803 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd24e1280a20445b8514fca298376f13 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cd24e1280a20445b8514fca298376f13" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd24e1280a20445b8514fca298376f13" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44875 } } }
I20260926 03:49:13.373767 21803 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd24e1280a20445b8514fca298376f13 [sys.catalog]: This master's current role is: LEADER
I20260926 03:49:13.375818 21809 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:49:13.381373 21809 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:49:13.390585 21809 catalog_manager.cc:1347] Generated new cluster ID: 3ab23a5128274d59aeb164e5360feed5
I20260926 03:49:13.390717 21809 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:49:13.411168 21809 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:49:13.411688 21809 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:49:13.424527 21809 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cd24e1280a20445b8514fca298376f13: Generated new TSK 0
I20260926 03:49:13.425686 21809 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 03:49:13.433405 779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu pid 21725
W20260926 03:49:13.434978 21787 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:51795
W20260926 03:49:13.435694 779 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu pid 21725: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 03:49:13.435849 779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu with pid 21725
2026-09-26T03:49:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1923 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790394221564982-779-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:49:13 dist-test-slave-n2th krb5kdc[21823](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:49:13 dist-test-slave-n2th krb5kdc[21823](info): set up 2 sockets
Sep 26 03:49:13 dist-test-slave-n2th krb5kdc[21823](info): commencing operation
krb5kdc: starting...
W20260926 03:49:14.551330 779 mini_kdc.cc:121] Time spent starting KDC: real 1.078s user 0.005s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790394221564982-779-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790394221564982-779-0/krb5kdc/test-user.keytab.
Sep 26 03:49:14 dist-test-slave-n2th krb5kdc[21823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394554, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:49:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:49:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790394221564982-779-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790394221564982-779-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 03:49:14.823387 779 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
/tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790394221564982-779-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790394221564982-779-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790394221564982-779-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790394221564982-779-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43363
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790394221564982-779-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43363
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790394221564982-779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790394221564982-779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790394221564982-779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:49:15.181902 21836 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:49:15.182288 21836 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:49:15.182468 21836 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:49:15.190110 21836 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:49:15.190285 21836 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:49:15.190400 21836 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:49:15.190507 21836 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:49:15.190591 21836 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:49:15.201071 21836 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790394221564982-779-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790394221564982-779-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43363
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790394221564982-779-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43363
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790394221564982-779-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790394221564982-779-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:49:15.202491 21836 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:49:15.203907 21836 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:49:15.216346 21841 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:49:15.216331 21844 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:49:15.216287 21842 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:49:15.217569 21836 server_base.cc:1034] running on GCE node
I20260926 03:49:15.218505 21836 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:49:15.220094 21836 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:49:15.221313 21836 hybrid_clock.cc:648] HybridClock initialized: now 1790394555221266 us; error 40 us; skew 500 ppm
I20260926 03:49:15.221695 21836 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:49:15 dist-test-slave-n2th krb5kdc[21823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394555, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:49:15.229598 21836 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 03:49:15.231504 21836 webserver.cc:460] Webserver started at http://127.0.0.1:46479/ using document root <none> and password file <none>
I20260926 03:49:15.232272 21836 fs_manager.cc:359] Metadata directory not provided
I20260926 03:49:15.232431 21836 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:49:15.232770 21836 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:49:15.236594 21836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790394221564982-779-0/minicluster-data/master-0/data/instance:
uuid: "11a1e338a15243a488cfe8b51d80d307"
format_stamp: "Formatted at 2026-09-26 03:49:15 on dist-test-slave-n2th"
server_key: "89c1ceee90d3e5fd8a4db35fc42f6e91"
server_key_iv: "1813a0911b89e81d7eef9f1d5611bf83"
server_key_version: "encryptionkey@0"
I20260926 03:49:15.237478 21836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790394221564982-779-0/minicluster-data/master-0/wal/instance:
uuid: "11a1e338a15243a488cfe8b51d80d307"
format_stamp: "Formatted at 2026-09-26 03:49:15 on dist-test-slave-n2th"
server_key: "89c1ceee90d3e5fd8a4db35fc42f6e91"
server_key_iv: "1813a0911b89e81d7eef9f1d5611bf83"
server_key_version: "encryptionkey@0"
I20260926 03:49:15.243453 21836 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260926 03:49:15.247545 21851 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:49:15.249084 21836 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 03:49:15.249311 21836 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790394221564982-779-0/minicluster-data/master-0/data,/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790394221564982-779-0/minicluster-data/master-0/wal
uuid: "11a1e338a15243a488cfe8b51d80d307"
format_stamp: "Formatted at 2026-09-26 03:49:15 on dist-test-slave-n2th"
server_key: "89c1ceee90d3e5fd8a4db35fc42f6e91"
server_key_iv: "1813a0911b89e81d7eef9f1d5611bf83"
server_key_version: "encryptionkey@0"
I20260926 03:49:15.249553 21836 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790394221564982-779-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790394221564982-779-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790394221564982-779-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:49:15.267752 21836 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:49:15.272744 21836 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:49:15.273092 21836 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:49:15.291733 21902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43363 every 8 connection(s)
I20260926 03:49:15.291760 21836 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43363
I20260926 03:49:15.293393 21836 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790394221564982-779-0/minicluster-data/master-0/data/info.pb
I20260926 03:49:15.297477 21903 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:49:15.302183 779 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu as pid 21836
I20260926 03:49:15.302393 779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790394221564982-779-0/minicluster-data/master-0/wal/instance
I20260926 03:49:15.302968 779 external_mini_cluster.cc:1428] Setting key a3ebe4c4baf9cfd7a0679975ee0544bb
I20260926 03:49:15.312132 21903 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 11a1e338a15243a488cfe8b51d80d307: Bootstrap starting.
I20260926 03:49:15.316320 21903 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 11a1e338a15243a488cfe8b51d80d307: Neither blocks nor log segments found. Creating new log.
I20260926 03:49:15.317771 21903 log.cc:824] T 00000000000000000000000000000000 P 11a1e338a15243a488cfe8b51d80d307: Log is configured to *not* fsync() on all Append() calls
Sep 26 03:49:15 dist-test-slave-n2th krb5kdc[21823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394554, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 03:49:15.321782 21903 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 11a1e338a15243a488cfe8b51d80d307: No bootstrap required, opened a new log
I20260926 03:49:15.329705 21906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:49:15.304956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.192:32995 (local address 127.0.0.1:43363)
0926 03:49:15.305768 (+ 812us) server_negotiation.cc:206] Beginning negotiation
0926 03:49:15.305780 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 03:49:15.305824 (+ 44us) server_negotiation.cc:407] Connection header received
0926 03:49:15.307642 (+ 1818us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:49:15.307671 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:49:15.308166 (+ 495us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:49:15.308534 (+ 368us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:49:15.310415 (+ 1881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:49:15.311894 (+ 1479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:49:15.313742 (+ 1848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:49:15.314524 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:49:15.314671 (+ 147us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:49:15.320057 (+ 5386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:49:15.320069 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:49:15.320090 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:49:15.320134 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:49:15.324317 (+ 4183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:49:15.325365 (+ 1048us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:49:15.325376 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:49:15.325387 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:49:15.325626 (+ 239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:49:15.326117 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:49:15.326122 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:49:15.326125 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:49:15.326937 (+ 812us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:49:15.327270 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0926 03:49:15.327765 (+ 495us) server_negotiation.cc:299] Negotiation successful
0926 03:49:15.328166 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":593,"thread_start_us":383,"threads_started":1}
I20260926 03:49:15.332808 21903 raft_consensus.cc:348] T 00000000000000000000000000000000 P 11a1e338a15243a488cfe8b51d80d307 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11a1e338a15243a488cfe8b51d80d307" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43363 } }
I20260926 03:49:15.333240 21903 raft_consensus.cc:374] T 00000000000000000000000000000000 P 11a1e338a15243a488cfe8b51d80d307 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:49:15.333374 21903 raft_consensus.cc:729] T 00000000000000000000000000000000 P 11a1e338a15243a488cfe8b51d80d307 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11a1e338a15243a488cfe8b51d80d307, State: Initialized, Role: FOLLOWER
I20260926 03:49:15.334118 21903 consensus_queue.cc:260] T 00000000000000000000000000000000 P 11a1e338a15243a488cfe8b51d80d307 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11a1e338a15243a488cfe8b51d80d307" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43363 } }
I20260926 03:49:15.334370 779 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 03:49:15.334450 21903 raft_consensus.cc:388] T 00000000000000000000000000000000 P 11a1e338a15243a488cfe8b51d80d307 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:49:15.334582 21903 raft_consensus.cc:482] T 00000000000000000000000000000000 P 11a1e338a15243a488cfe8b51d80d307 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:49:15.334766 21903 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 11a1e338a15243a488cfe8b51d80d307 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:49:15.338128 21903 raft_consensus.cc:504] T 00000000000000000000000000000000 P 11a1e338a15243a488cfe8b51d80d307 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11a1e338a15243a488cfe8b51d80d307" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43363 } }
I20260926 03:49:15.338867 21903 leader_election.cc:302] T 00000000000000000000000000000000 P 11a1e338a15243a488cfe8b51d80d307 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 11a1e338a15243a488cfe8b51d80d307; no voters:
I20260926 03:49:15.339543 21903 leader_election.cc:288] T 00000000000000000000000000000000 P 11a1e338a15243a488cfe8b51d80d307 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:49:15.339718 21912 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 11a1e338a15243a488cfe8b51d80d307 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:49:15.343145 21912 raft_consensus.cc:686] T 00000000000000000000000000000000 P 11a1e338a15243a488cfe8b51d80d307 [term 1 LEADER]: Becoming Leader. State: Replica: 11a1e338a15243a488cfe8b51d80d307, State: Running, Role: LEADER
I20260926 03:49:15.344148 21912 consensus_queue.cc:237] T 00000000000000000000000000000000 P 11a1e338a15243a488cfe8b51d80d307 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11a1e338a15243a488cfe8b51d80d307" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43363 } }
I20260926 03:49:15.345777 21903 sys_catalog.cc:564] T 00000000000000000000000000000000 P 11a1e338a15243a488cfe8b51d80d307 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:49:15.347429 21915 sys_catalog.cc:455] T 00000000000000000000000000000000 P 11a1e338a15243a488cfe8b51d80d307 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "11a1e338a15243a488cfe8b51d80d307" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11a1e338a15243a488cfe8b51d80d307" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43363 } } }
I20260926 03:49:15.348011 21915 sys_catalog.cc:458] T 00000000000000000000000000000000 P 11a1e338a15243a488cfe8b51d80d307 [sys.catalog]: This master's current role is: LEADER
I20260926 03:49:15.348658 21913 sys_catalog.cc:455] T 00000000000000000000000000000000 P 11a1e338a15243a488cfe8b51d80d307 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 11a1e338a15243a488cfe8b51d80d307. Latest consensus state: current_term: 1 leader_uuid: "11a1e338a15243a488cfe8b51d80d307" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11a1e338a15243a488cfe8b51d80d307" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43363 } } }
I20260926 03:49:15.349015 21913 sys_catalog.cc:458] T 00000000000000000000000000000000 P 11a1e338a15243a488cfe8b51d80d307 [sys.catalog]: This master's current role is: LEADER
I20260926 03:49:15.351307 21918 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:49:15.359741 21918 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:49:15.363617 21906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:49:15.339977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.192:46985 (local address 127.0.0.1:43363)
0926 03:49:15.340229 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0926 03:49:15.340239 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 03:49:15.344775 (+ 4536us) server_negotiation.cc:407] Connection header received
0926 03:49:15.345052 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:49:15.345058 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:49:15.345141 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:49:15.345255 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:49:15.347941 (+ 2686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:49:15.349061 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:49:15.350715 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:49:15.351518 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:49:15.351670 (+ 152us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:49:15.355288 (+ 3618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:49:15.355295 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:49:15.355299 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:49:15.355361 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:49:15.358533 (+ 3172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:49:15.359866 (+ 1333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:49:15.359875 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:49:15.359882 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:49:15.360081 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:49:15.360864 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:49:15.360873 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:49:15.360879 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:49:15.361441 (+ 562us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:49:15.361789 (+ 348us) server_negotiation.cc:1035] Waiting for connection context
0926 03:49:15.362703 (+ 914us) server_negotiation.cc:299] Negotiation successful
0926 03:49:15.363028 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260926 03:49:15.370759 21918 catalog_manager.cc:1347] Generated new cluster ID: 165c8d56ac64438797a00c2e38ab52d1
I20260926 03:49:15.370926 21918 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:49:15.393035 21918 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:49:15.395020 21918 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:49:15.412451 21918 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 11a1e338a15243a488cfe8b51d80d307: Generated new TSK 0
I20260926 03:49:15.413367 21918 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 03:49:15.423007 779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu pid 21836
I20260926 03:49:15.424335 21898 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:49:15.518545 779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu with pid 21836
W20260926 03:49:15.533428 21821 connection.cc:535] client connection to 127.0.0.1:43363 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43363): Connection reset by peer (error 104)
2026-09-26T03:49:15Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (2080 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790394221564982-779-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:49:15 dist-test-slave-n2th krb5kdc[21935](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:49:15 dist-test-slave-n2th krb5kdc[21935](info): set up 2 sockets
Sep 26 03:49:15 dist-test-slave-n2th krb5kdc[21935](info): commencing operation
krb5kdc: starting...
W20260926 03:49:16.631255 779 mini_kdc.cc:121] Time spent starting KDC: real 1.078s user 0.007s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790394221564982-779-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790394221564982-779-0/krb5kdc/test-user.keytab.
Sep 26 03:49:16 dist-test-slave-n2th krb5kdc[21935](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394556, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:49:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:49:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790394221564982-779-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790394221564982-779-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 03:49:16.914147 779 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
/tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790394221564982-779-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790394221564982-779-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790394221564982-779-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790394221564982-779-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45171
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790394221564982-779-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42901
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45171
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790394221564982-779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790394221564982-779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790394221564982-779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:49:17.250955 21948 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:49:17.251448 21948 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:49:17.251627 21948 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:49:17.259043 21948 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:49:17.259239 21948 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:49:17.259327 21948 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:49:17.259421 21948 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:49:17.259512 21948 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:49:17.269582 21948 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42901
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790394221564982-779-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790394221564982-779-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45171
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790394221564982-779-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45171
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790394221564982-779-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790394221564982-779-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:49:17.271045 21948 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:49:17.272387 21948 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:49:17.285072 21953 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:49:17.285010 21956 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:49:17.285010 21954 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:49:17.286211 21948 server_base.cc:1034] running on GCE node
I20260926 03:49:17.287344 21948 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:49:17.288897 21948 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:49:17.290143 21948 hybrid_clock.cc:648] HybridClock initialized: now 1790394557290119 us; error 63 us; skew 500 ppm
I20260926 03:49:17.290514 21948 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:49:17 dist-test-slave-n2th krb5kdc[21935](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394557, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:49:17.293960 21948 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 03:49:17.295732 21948 webserver.cc:460] Webserver started at http://127.0.0.1:42345/ using document root <none> and password file <none>
I20260926 03:49:17.296486 21948 fs_manager.cc:359] Metadata directory not provided
I20260926 03:49:17.296644 21948 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:49:17.296988 21948 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:49:17.300366 21948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790394221564982-779-0/minicluster-data/master-0/data/instance:
uuid: "9e4c840436814fb0b6b5aab6227a18d2"
format_stamp: "Formatted at 2026-09-26 03:49:17 on dist-test-slave-n2th"
server_key: "7d44e861fbb50d34056ae3c50d77fec6"
server_key_iv: "eeec9456c6dafa628348f0430c4f3be3"
server_key_version: "encryptionkey@0"
I20260926 03:49:17.301220 21948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790394221564982-779-0/minicluster-data/master-0/wal/instance:
uuid: "9e4c840436814fb0b6b5aab6227a18d2"
format_stamp: "Formatted at 2026-09-26 03:49:17 on dist-test-slave-n2th"
server_key: "7d44e861fbb50d34056ae3c50d77fec6"
server_key_iv: "eeec9456c6dafa628348f0430c4f3be3"
server_key_version: "encryptionkey@0"
I20260926 03:49:17.307001 21948 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260926 03:49:17.310706 21963 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:49:17.312284 21948 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 03:49:17.312502 21948 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790394221564982-779-0/minicluster-data/master-0/data,/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790394221564982-779-0/minicluster-data/master-0/wal
uuid: "9e4c840436814fb0b6b5aab6227a18d2"
format_stamp: "Formatted at 2026-09-26 03:49:17 on dist-test-slave-n2th"
server_key: "7d44e861fbb50d34056ae3c50d77fec6"
server_key_iv: "eeec9456c6dafa628348f0430c4f3be3"
server_key_version: "encryptionkey@0"
I20260926 03:49:17.312742 21948 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790394221564982-779-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790394221564982-779-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790394221564982-779-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:49:17.345392 21948 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:49:17.350085 21948 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:49:17.350390 21948 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:49:17.369653 21948 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45171
I20260926 03:49:17.369688 22014 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45171 every 8 connection(s)
I20260926 03:49:17.371201 21948 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790394221564982-779-0/minicluster-data/master-0/data/info.pb
I20260926 03:49:17.372560 779 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu as pid 21948
I20260926 03:49:17.372762 779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790394221564982-779-0/minicluster-data/master-0/wal/instance
I20260926 03:49:17.373315 779 external_mini_cluster.cc:1428] Setting key 576ec24bd19f271e2f40c9ef275dd4ec
I20260926 03:49:17.377084 22015 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:49:17.386081 22018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:49:17.375087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.192:38003 (local address 127.0.0.1:45171)
0926 03:49:17.375736 (+ 649us) server_negotiation.cc:206] Beginning negotiation
0926 03:49:17.375748 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 03:49:17.375790 (+ 42us) server_negotiation.cc:407] Connection header received
0926 03:49:17.377529 (+ 1739us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:49:17.377552 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:49:17.377931 (+ 379us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:49:17.378313 (+ 382us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:49:17.378889 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:49:17.380214 (+ 1325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:49:17.381607 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:49:17.382305 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:49:17.382509 (+ 204us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:49:17.383805 (+ 1296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:49:17.383812 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:49:17.383825 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:49:17.384373 (+ 548us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:49:17.384445 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:49:17.384476 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:49:17.384674 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0926 03:49:17.384943 (+ 269us) server_negotiation.cc:299] Negotiation successful
0926 03:49:17.385112 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":459,"thread_start_us":260,"threads_started":1}
I20260926 03:49:17.387897 22015 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e4c840436814fb0b6b5aab6227a18d2: Bootstrap starting.
I20260926 03:49:17.390010 779 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 03:49:17.391474 22015 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9e4c840436814fb0b6b5aab6227a18d2: Neither blocks nor log segments found. Creating new log.
I20260926 03:49:17.392721 22015 log.cc:824] T 00000000000000000000000000000000 P 9e4c840436814fb0b6b5aab6227a18d2: Log is configured to *not* fsync() on all Append() calls
I20260926 03:49:17.396055 22015 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e4c840436814fb0b6b5aab6227a18d2: No bootstrap required, opened a new log
I20260926 03:49:17.402551 22018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:49:17.395503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.192:47113 (local address 127.0.0.1:45171)
0926 03:49:17.395697 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0926 03:49:17.395703 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:49:17.396198 (+ 495us) server_negotiation.cc:407] Connection header received
0926 03:49:17.396390 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:49:17.396395 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:49:17.396472 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:49:17.396616 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:49:17.397103 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:49:17.398298 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:49:17.399718 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:49:17.400433 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:49:17.400565 (+ 132us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:49:17.401689 (+ 1124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:49:17.401697 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:49:17.401706 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:49:17.401730 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:49:17.401779 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:49:17.401804 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:49:17.401919 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0926 03:49:17.402117 (+ 198us) server_negotiation.cc:299] Negotiation successful
0926 03:49:17.402178 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 03:49:17.405303 22015 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9e4c840436814fb0b6b5aab6227a18d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e4c840436814fb0b6b5aab6227a18d2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45171 } }
I20260926 03:49:17.405705 22015 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9e4c840436814fb0b6b5aab6227a18d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:49:17.405831 22015 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9e4c840436814fb0b6b5aab6227a18d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e4c840436814fb0b6b5aab6227a18d2, State: Initialized, Role: FOLLOWER
I20260926 03:49:17.406594 22015 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9e4c840436814fb0b6b5aab6227a18d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e4c840436814fb0b6b5aab6227a18d2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45171 } }
I20260926 03:49:17.407008 22015 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9e4c840436814fb0b6b5aab6227a18d2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:49:17.407184 22015 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9e4c840436814fb0b6b5aab6227a18d2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:49:17.407385 22015 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9e4c840436814fb0b6b5aab6227a18d2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:49:17.409744 22015 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9e4c840436814fb0b6b5aab6227a18d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e4c840436814fb0b6b5aab6227a18d2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45171 } }
I20260926 03:49:17.410300 22015 leader_election.cc:302] T 00000000000000000000000000000000 P 9e4c840436814fb0b6b5aab6227a18d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9e4c840436814fb0b6b5aab6227a18d2; no voters:
I20260926 03:49:17.410985 22015 leader_election.cc:288] T 00000000000000000000000000000000 P 9e4c840436814fb0b6b5aab6227a18d2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:49:17.411151 22025 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9e4c840436814fb0b6b5aab6227a18d2 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:49:17.412338 22025 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9e4c840436814fb0b6b5aab6227a18d2 [term 1 LEADER]: Becoming Leader. State: Replica: 9e4c840436814fb0b6b5aab6227a18d2, State: Running, Role: LEADER
I20260926 03:49:17.412858 22025 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9e4c840436814fb0b6b5aab6227a18d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e4c840436814fb0b6b5aab6227a18d2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45171 } }
I20260926 03:49:17.413339 22015 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9e4c840436814fb0b6b5aab6227a18d2 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:49:17.415943 22027 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e4c840436814fb0b6b5aab6227a18d2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9e4c840436814fb0b6b5aab6227a18d2. Latest consensus state: current_term: 1 leader_uuid: "9e4c840436814fb0b6b5aab6227a18d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e4c840436814fb0b6b5aab6227a18d2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45171 } } }
I20260926 03:49:17.416445 22027 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e4c840436814fb0b6b5aab6227a18d2 [sys.catalog]: This master's current role is: LEADER
I20260926 03:49:17.417014 22026 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e4c840436814fb0b6b5aab6227a18d2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9e4c840436814fb0b6b5aab6227a18d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e4c840436814fb0b6b5aab6227a18d2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45171 } } }
I20260926 03:49:17.417379 22026 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e4c840436814fb0b6b5aab6227a18d2 [sys.catalog]: This master's current role is: LEADER
I20260926 03:49:17.420845 22032 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:49:17.425679 22032 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:49:17.434307 22032 catalog_manager.cc:1347] Generated new cluster ID: 3fe3d115406f47f49d67f620021f89a2
I20260926 03:49:17.434448 22032 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:49:17.452831 22032 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:49:17.454766 22032 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:49:17.465135 22032 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9e4c840436814fb0b6b5aab6227a18d2: Generated new TSK 0
I20260926 03:49:17.466271 22032 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 03:49:17.475760 779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu pid 21948
W20260926 03:49:17.477101 22010 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.192:38003
W20260926 03:49:17.477924 779 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu pid 21948: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 03:49:17.478124 779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu with pid 21948
W20260926 03:49:17.495491 21933 connection.cc:535] client connection to 127.0.0.1:45171 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45171): Connection reset by peer (error 104)
2026-09-26T03:49:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1962 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790394221564982-779-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:49:17 dist-test-slave-n2th krb5kdc[22046](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:49:17 dist-test-slave-n2th krb5kdc[22046](info): set up 2 sockets
Sep 26 03:49:17 dist-test-slave-n2th krb5kdc[22046](info): commencing operation
krb5kdc: starting...
W20260926 03:49:18.596529 779 mini_kdc.cc:121] Time spent starting KDC: real 1.081s user 0.001s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790394221564982-779-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790394221564982-779-0/krb5kdc/test-user.keytab.
Sep 26 03:49:18 dist-test-slave-n2th krb5kdc[22046](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394558, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:49:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:49:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790394221564982-779-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790394221564982-779-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 03:49:18.871521 779 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
/tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790394221564982-779-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790394221564982-779-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790394221564982-779-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790394221564982-779-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44419
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790394221564982-779-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44419
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790394221564982-779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790394221564982-779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790394221564982-779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:49:19.211964 22059 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:49:19.212282 22059 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:49:19.212489 22059 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:49:19.219918 22059 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:49:19.220038 22059 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:49:19.220158 22059 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:49:19.220274 22059 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:49:19.220355 22059 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:49:19.230577 22059 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790394221564982-779-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790394221564982-779-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44419
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790394221564982-779-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44419
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790394221564982-779-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790394221564982-779-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:49:19.231966 22059 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:49:19.233305 22059 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:49:19.245736 22064 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:49:19.245810 22065 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:49:19.245760 22067 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:49:19.246577 22059 server_base.cc:1034] running on GCE node
I20260926 03:49:19.247501 22059 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:49:19.248960 22059 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:49:19.250218 22059 hybrid_clock.cc:648] HybridClock initialized: now 1790394559250177 us; error 41 us; skew 500 ppm
I20260926 03:49:19.250598 22059 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:49:19 dist-test-slave-n2th krb5kdc[22046](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394559, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:49:19.254037 22059 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 03:49:19.255767 22059 webserver.cc:460] Webserver started at http://127.0.0.1:37035/ using document root <none> and password file <none>
I20260926 03:49:19.256510 22059 fs_manager.cc:359] Metadata directory not provided
I20260926 03:49:19.256669 22059 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:49:19.257068 22059 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:49:19.260309 22059 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790394221564982-779-0/minicluster-data/master-0/data/instance:
uuid: "a6c290c9c426439cbe4622d01f98d655"
format_stamp: "Formatted at 2026-09-26 03:49:19 on dist-test-slave-n2th"
server_key: "b6b6734ac1365a5e93c9c407c171a4a6"
server_key_iv: "2770df380b7375520fd862b3b385b3bb"
server_key_version: "encryptionkey@0"
I20260926 03:49:19.261231 22059 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790394221564982-779-0/minicluster-data/master-0/wal/instance:
uuid: "a6c290c9c426439cbe4622d01f98d655"
format_stamp: "Formatted at 2026-09-26 03:49:19 on dist-test-slave-n2th"
server_key: "b6b6734ac1365a5e93c9c407c171a4a6"
server_key_iv: "2770df380b7375520fd862b3b385b3bb"
server_key_version: "encryptionkey@0"
I20260926 03:49:19.266909 22059 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260926 03:49:19.270718 22074 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:49:19.272295 22059 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 03:49:19.272531 22059 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790394221564982-779-0/minicluster-data/master-0/data,/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790394221564982-779-0/minicluster-data/master-0/wal
uuid: "a6c290c9c426439cbe4622d01f98d655"
format_stamp: "Formatted at 2026-09-26 03:49:19 on dist-test-slave-n2th"
server_key: "b6b6734ac1365a5e93c9c407c171a4a6"
server_key_iv: "2770df380b7375520fd862b3b385b3bb"
server_key_version: "encryptionkey@0"
I20260926 03:49:19.272778 22059 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790394221564982-779-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790394221564982-779-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790394221564982-779-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:49:19.279078 22059 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:49:19.283293 22059 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:49:19.283604 22059 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:49:19.302932 22059 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44419
I20260926 03:49:19.302963 22125 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44419 every 8 connection(s)
I20260926 03:49:19.304425 22059 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790394221564982-779-0/minicluster-data/master-0/data/info.pb
I20260926 03:49:19.308271 779 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu as pid 22059
I20260926 03:49:19.308380 22126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:49:19.308516 779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790394221564982-779-0/minicluster-data/master-0/wal/instance
I20260926 03:49:19.308948 779 external_mini_cluster.cc:1428] Setting key 9c9c5960eb1c7074b9e3ee2deb5b8e8c
I20260926 03:49:19.317708 22129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:49:19.311063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.192:49215 (local address 127.0.0.1:44419)
0926 03:49:19.311950 (+ 887us) server_negotiation.cc:206] Beginning negotiation
0926 03:49:19.311962 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 03:49:19.312011 (+ 49us) server_negotiation.cc:407] Connection header received
0926 03:49:19.313865 (+ 1854us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:49:19.313891 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:49:19.314294 (+ 403us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:49:19.314847 (+ 553us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:49:19.315994 (+ 1147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:49:19.316000 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:49:19.316017 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:49:19.316040 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:49:19.316097 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:49:19.316120 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:49:19.316397 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0926 03:49:19.316550 (+ 153us) server_negotiation.cc:299] Negotiation successful
0926 03:49:19.316746 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":619,"thread_start_us":350,"threads_started":1}
I20260926 03:49:19.321846 779 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 03:49:19.323038 22126 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6c290c9c426439cbe4622d01f98d655: Bootstrap starting.
I20260926 03:49:19.326304 22126 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a6c290c9c426439cbe4622d01f98d655: Neither blocks nor log segments found. Creating new log.
I20260926 03:49:19.327844 22126 log.cc:824] T 00000000000000000000000000000000 P a6c290c9c426439cbe4622d01f98d655: Log is configured to *not* fsync() on all Append() calls
I20260926 03:49:19.329630 22129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:49:19.326788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.192:38557 (local address 127.0.0.1:44419)
0926 03:49:19.327015 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0926 03:49:19.327022 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:49:19.327378 (+ 356us) server_negotiation.cc:407] Connection header received
0926 03:49:19.327525 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:49:19.327533 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:49:19.327613 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:49:19.327739 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:49:19.328878 (+ 1139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:49:19.328883 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:49:19.328887 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:49:19.328908 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:49:19.328947 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:49:19.328962 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:49:19.329061 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0926 03:49:19.329220 (+ 159us) server_negotiation.cc:299] Negotiation successful
0926 03:49:19.329271 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260926 03:49:19.331290 22126 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6c290c9c426439cbe4622d01f98d655: No bootstrap required, opened a new log
I20260926 03:49:19.339303 22126 raft_consensus.cc:348] T 00000000000000000000000000000000 P a6c290c9c426439cbe4622d01f98d655 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6c290c9c426439cbe4622d01f98d655" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44419 } }
I20260926 03:49:19.339733 22126 raft_consensus.cc:374] T 00000000000000000000000000000000 P a6c290c9c426439cbe4622d01f98d655 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:49:19.339867 22126 raft_consensus.cc:729] T 00000000000000000000000000000000 P a6c290c9c426439cbe4622d01f98d655 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6c290c9c426439cbe4622d01f98d655, State: Initialized, Role: FOLLOWER
I20260926 03:49:19.340471 22126 consensus_queue.cc:260] T 00000000000000000000000000000000 P a6c290c9c426439cbe4622d01f98d655 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6c290c9c426439cbe4622d01f98d655" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44419 } }
I20260926 03:49:19.340714 22126 raft_consensus.cc:388] T 00000000000000000000000000000000 P a6c290c9c426439cbe4622d01f98d655 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:49:19.340894 22126 raft_consensus.cc:482] T 00000000000000000000000000000000 P a6c290c9c426439cbe4622d01f98d655 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:49:19.341064 22126 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a6c290c9c426439cbe4622d01f98d655 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:49:19.343657 22126 raft_consensus.cc:504] T 00000000000000000000000000000000 P a6c290c9c426439cbe4622d01f98d655 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6c290c9c426439cbe4622d01f98d655" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44419 } }
I20260926 03:49:19.344213 22126 leader_election.cc:302] T 00000000000000000000000000000000 P a6c290c9c426439cbe4622d01f98d655 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a6c290c9c426439cbe4622d01f98d655; no voters:
I20260926 03:49:19.344831 22126 leader_election.cc:288] T 00000000000000000000000000000000 P a6c290c9c426439cbe4622d01f98d655 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:49:19.344993 22136 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a6c290c9c426439cbe4622d01f98d655 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:49:19.345713 22136 raft_consensus.cc:686] T 00000000000000000000000000000000 P a6c290c9c426439cbe4622d01f98d655 [term 1 LEADER]: Becoming Leader. State: Replica: a6c290c9c426439cbe4622d01f98d655, State: Running, Role: LEADER
I20260926 03:49:19.346213 22136 consensus_queue.cc:237] T 00000000000000000000000000000000 P a6c290c9c426439cbe4622d01f98d655 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6c290c9c426439cbe4622d01f98d655" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44419 } }
I20260926 03:49:19.347123 22126 sys_catalog.cc:564] T 00000000000000000000000000000000 P a6c290c9c426439cbe4622d01f98d655 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:49:19.349504 22138 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6c290c9c426439cbe4622d01f98d655 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a6c290c9c426439cbe4622d01f98d655. Latest consensus state: current_term: 1 leader_uuid: "a6c290c9c426439cbe4622d01f98d655" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6c290c9c426439cbe4622d01f98d655" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44419 } } }
I20260926 03:49:19.349941 22138 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6c290c9c426439cbe4622d01f98d655 [sys.catalog]: This master's current role is: LEADER
I20260926 03:49:19.351408 22137 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6c290c9c426439cbe4622d01f98d655 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a6c290c9c426439cbe4622d01f98d655" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6c290c9c426439cbe4622d01f98d655" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44419 } } }
I20260926 03:49:19.351866 22137 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6c290c9c426439cbe4622d01f98d655 [sys.catalog]: This master's current role is: LEADER
I20260926 03:49:19.356038 22141 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:49:19.363705 22141 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:49:19.373574 22141 catalog_manager.cc:1347] Generated new cluster ID: e2d2ec7885374d0cb8cee58a27cf918e
I20260926 03:49:19.373782 22141 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:49:19.403098 22141 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:49:19.403620 22141 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:49:19.417541 22141 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a6c290c9c426439cbe4622d01f98d655: Generated new TSK 0
I20260926 03:49:19.418471 22141 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 03:49:19.430456 779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu pid 22059
W20260926 03:49:19.431735 22121 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.192:49215
W20260926 03:49:19.432479 779 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu pid 22059: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 03:49:19.432653 779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu with pid 22059
2026-09-26T03:49:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1953 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790394221564982-779-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:49:19 dist-test-slave-n2th krb5kdc[22157](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:49:19 dist-test-slave-n2th krb5kdc[22157](info): set up 2 sockets
Sep 26 03:49:19 dist-test-slave-n2th krb5kdc[22157](info): commencing operation
krb5kdc: starting...
W20260926 03:49:20.547254 779 mini_kdc.cc:121] Time spent starting KDC: real 1.078s user 0.006s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790394221564982-779-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790394221564982-779-0/krb5kdc/test-user.keytab.
Sep 26 03:49:20 dist-test-slave-n2th krb5kdc[22157](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394560, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:49:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:49:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790394221564982-779-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790394221564982-779-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 03:49:20.818353 779 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
/tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790394221564982-779-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790394221564982-779-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790394221564982-779-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790394221564982-779-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32843
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790394221564982-779-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:32843
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790394221564982-779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790394221564982-779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790394221564982-779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:49:21.163436 22170 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:49:21.163738 22170 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:49:21.163936 22170 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:49:21.171523 22170 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:49:21.171650 22170 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:49:21.171798 22170 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:49:21.171918 22170 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:49:21.172011 22170 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:49:21.182155 22170 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790394221564982-779-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790394221564982-779-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:32843
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790394221564982-779-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32843
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790394221564982-779-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790394221564982-779-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:49:21.183584 22170 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:49:21.184922 22170 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:49:21.197255 22175 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:49:21.197186 22176 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:49:21.197192 22178 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:49:21.198297 22170 server_base.cc:1034] running on GCE node
I20260926 03:49:21.199218 22170 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:49:21.200814 22170 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:49:21.202054 22170 hybrid_clock.cc:648] HybridClock initialized: now 1790394561202012 us; error 40 us; skew 500 ppm
I20260926 03:49:21.202419 22170 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:49:21 dist-test-slave-n2th krb5kdc[22157](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394561, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:49:21.212237 22170 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 03:49:21.213987 22170 webserver.cc:460] Webserver started at http://127.0.0.1:41421/ using document root <none> and password file <none>
I20260926 03:49:21.214824 22170 fs_manager.cc:359] Metadata directory not provided
I20260926 03:49:21.214982 22170 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:49:21.215310 22170 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:49:21.218544 22170 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790394221564982-779-0/minicluster-data/master-0/data/instance:
uuid: "8c8afecae0034402879c57a9baf22779"
format_stamp: "Formatted at 2026-09-26 03:49:21 on dist-test-slave-n2th"
server_key: "a389dee1e83fdb7808ef925e4ea3806a"
server_key_iv: "a2a6dacbcfd5cfe54bdceba8975ea8a0"
server_key_version: "encryptionkey@0"
I20260926 03:49:21.219420 22170 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790394221564982-779-0/minicluster-data/master-0/wal/instance:
uuid: "8c8afecae0034402879c57a9baf22779"
format_stamp: "Formatted at 2026-09-26 03:49:21 on dist-test-slave-n2th"
server_key: "a389dee1e83fdb7808ef925e4ea3806a"
server_key_iv: "a2a6dacbcfd5cfe54bdceba8975ea8a0"
server_key_version: "encryptionkey@0"
I20260926 03:49:21.224670 22170 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260926 03:49:21.228677 22185 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:49:21.230043 22170 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 03:49:21.230254 22170 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790394221564982-779-0/minicluster-data/master-0/data,/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790394221564982-779-0/minicluster-data/master-0/wal
uuid: "8c8afecae0034402879c57a9baf22779"
format_stamp: "Formatted at 2026-09-26 03:49:21 on dist-test-slave-n2th"
server_key: "a389dee1e83fdb7808ef925e4ea3806a"
server_key_iv: "a2a6dacbcfd5cfe54bdceba8975ea8a0"
server_key_version: "encryptionkey@0"
I20260926 03:49:21.230492 22170 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790394221564982-779-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790394221564982-779-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790394221564982-779-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:49:21.267922 22170 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:49:21.272169 22170 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:49:21.272485 22170 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:49:21.291903 22170 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:32843
I20260926 03:49:21.291927 22236 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:32843 every 8 connection(s)
I20260926 03:49:21.293390 22170 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790394221564982-779-0/minicluster-data/master-0/data/info.pb
I20260926 03:49:21.295706 779 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu as pid 22170
I20260926 03:49:21.295909 779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790394221564982-779-0/minicluster-data/master-0/wal/instance
I20260926 03:49:21.296459 779 external_mini_cluster.cc:1428] Setting key 89a3f4cbc215f15222c5b8746489aa40
I20260926 03:49:21.297973 22237 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 26 03:49:21 dist-test-slave-n2th krb5kdc[22157](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394560, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 03:49:21.310226 22237 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c8afecae0034402879c57a9baf22779: Bootstrap starting.
I20260926 03:49:21.313877 22237 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8c8afecae0034402879c57a9baf22779: Neither blocks nor log segments found. Creating new log.
I20260926 03:49:21.315073 22237 log.cc:824] T 00000000000000000000000000000000 P 8c8afecae0034402879c57a9baf22779: Log is configured to *not* fsync() on all Append() calls
I20260926 03:49:21.318176 22237 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c8afecae0034402879c57a9baf22779: No bootstrap required, opened a new log
I20260926 03:49:21.319061 22240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:49:21.298383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.192:35471 (local address 127.0.0.1:32843)
0926 03:49:21.299228 (+ 845us) server_negotiation.cc:206] Beginning negotiation
0926 03:49:21.299240 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 03:49:21.299282 (+ 42us) server_negotiation.cc:407] Connection header received
0926 03:49:21.300910 (+ 1628us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:49:21.300933 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:49:21.301371 (+ 438us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:49:21.301803 (+ 432us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:49:21.303417 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:49:21.304743 (+ 1326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:49:21.306015 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:49:21.306718 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:49:21.307056 (+ 338us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:49:21.311374 (+ 4318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:49:21.311382 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:49:21.311397 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:49:21.311442 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:49:21.314611 (+ 3169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:49:21.315528 (+ 917us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:49:21.315535 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:49:21.315541 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:49:21.315719 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:49:21.316219 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:49:21.316223 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:49:21.316225 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:49:21.316737 (+ 512us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:49:21.317079 (+ 342us) server_negotiation.cc:1035] Waiting for connection context
0926 03:49:21.317575 (+ 496us) server_negotiation.cc:299] Negotiation successful
0926 03:49:21.317886 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":630,"thread_start_us":394,"threads_started":1}
I20260926 03:49:21.323096 779 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 03:49:21.326895 22237 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8c8afecae0034402879c57a9baf22779 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c8afecae0034402879c57a9baf22779" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32843 } }
I20260926 03:49:21.327356 22237 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8c8afecae0034402879c57a9baf22779 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:49:21.327485 22237 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8c8afecae0034402879c57a9baf22779 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c8afecae0034402879c57a9baf22779, State: Initialized, Role: FOLLOWER
I20260926 03:49:21.328238 22237 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8c8afecae0034402879c57a9baf22779 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c8afecae0034402879c57a9baf22779" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32843 } }
I20260926 03:49:21.328540 22237 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8c8afecae0034402879c57a9baf22779 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:49:21.328715 22237 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8c8afecae0034402879c57a9baf22779 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:49:21.328930 22237 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8c8afecae0034402879c57a9baf22779 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:49:21.331323 22237 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8c8afecae0034402879c57a9baf22779 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c8afecae0034402879c57a9baf22779" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32843 } }
I20260926 03:49:21.331866 22237 leader_election.cc:302] T 00000000000000000000000000000000 P 8c8afecae0034402879c57a9baf22779 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8c8afecae0034402879c57a9baf22779; no voters:
I20260926 03:49:21.332518 22237 leader_election.cc:288] T 00000000000000000000000000000000 P 8c8afecae0034402879c57a9baf22779 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:49:21.332672 22247 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8c8afecae0034402879c57a9baf22779 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:49:21.335407 22247 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8c8afecae0034402879c57a9baf22779 [term 1 LEADER]: Becoming Leader. State: Replica: 8c8afecae0034402879c57a9baf22779, State: Running, Role: LEADER
I20260926 03:49:21.336006 22237 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8c8afecae0034402879c57a9baf22779 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:49:21.336002 22247 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8c8afecae0034402879c57a9baf22779 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c8afecae0034402879c57a9baf22779" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32843 } }
I20260926 03:49:21.339447 22249 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c8afecae0034402879c57a9baf22779 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8c8afecae0034402879c57a9baf22779. Latest consensus state: current_term: 1 leader_uuid: "8c8afecae0034402879c57a9baf22779" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c8afecae0034402879c57a9baf22779" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32843 } } }
I20260926 03:49:21.339919 22249 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c8afecae0034402879c57a9baf22779 [sys.catalog]: This master's current role is: LEADER
I20260926 03:49:21.340389 22248 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c8afecae0034402879c57a9baf22779 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8c8afecae0034402879c57a9baf22779" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c8afecae0034402879c57a9baf22779" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32843 } } }
I20260926 03:49:21.340752 22248 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c8afecae0034402879c57a9baf22779 [sys.catalog]: This master's current role is: LEADER
I20260926 03:49:21.345006 22256 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:49:21.348111 22240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:49:21.327551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.192:49239 (local address 127.0.0.1:32843)
0926 03:49:21.327732 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0926 03:49:21.327737 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:49:21.328436 (+ 699us) server_negotiation.cc:407] Connection header received
0926 03:49:21.328687 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:49:21.328693 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:49:21.328771 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:49:21.328894 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:49:21.330487 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:49:21.331357 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:49:21.332931 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:49:21.334142 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:49:21.334301 (+ 159us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:49:21.337763 (+ 3462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:49:21.337771 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:49:21.337779 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:49:21.337831 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:49:21.342281 (+ 4450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:49:21.343821 (+ 1540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:49:21.343829 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:49:21.343834 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:49:21.344082 (+ 248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:49:21.344826 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:49:21.344833 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:49:21.344837 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:49:21.345335 (+ 498us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:49:21.347048 (+ 1713us) server_negotiation.cc:1035] Waiting for connection context
0926 03:49:21.347288 (+ 240us) server_negotiation.cc:299] Negotiation successful
0926 03:49:21.347547 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 03:49:21.352684 22256 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:49:21.363161 22256 catalog_manager.cc:1347] Generated new cluster ID: 4e10772939b94d1b858bda72fe7ae9a6
I20260926 03:49:21.363291 22256 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:49:21.377584 22256 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:49:21.379586 22256 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:49:21.392484 22256 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8c8afecae0034402879c57a9baf22779: Generated new TSK 0
I20260926 03:49:21.393373 22256 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 03:49:21.407577 779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu pid 22170
I20260926 03:49:21.409297 22232 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:49:21.502310 779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu with pid 22170
W20260926 03:49:21.519855 22155 connection.cc:535] client connection to 127.0.0.1:32843 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:32843): Connection reset by peer (error 104)
2026-09-26T03:49:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (2070 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790394221564982-779-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:49:21 dist-test-slave-n2th krb5kdc[22270](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:49:21 dist-test-slave-n2th krb5kdc[22270](info): set up 2 sockets
Sep 26 03:49:21 dist-test-slave-n2th krb5kdc[22270](info): commencing operation
krb5kdc: starting...
W20260926 03:49:22.627228 779 mini_kdc.cc:121] Time spent starting KDC: real 1.088s user 0.004s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790394221564982-779-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790394221564982-779-0/krb5kdc/test-user.keytab.
Sep 26 03:49:22 dist-test-slave-n2th krb5kdc[22270](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394562, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:49:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:49:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790394221564982-779-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790394221564982-779-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 03:49:22.898540 779 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
/tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790394221564982-779-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790394221564982-779-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790394221564982-779-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790394221564982-779-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46753
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790394221564982-779-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46753
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790394221564982-779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790394221564982-779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790394221564982-779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:49:23.240792 22283 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:49:23.241182 22283 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:49:23.241350 22283 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:49:23.248957 22283 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:49:23.249153 22283 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:49:23.249264 22283 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:49:23.249382 22283 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:49:23.249506 22283 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:49:23.260228 22283 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790394221564982-779-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790394221564982-779-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46753
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790394221564982-779-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46753
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790394221564982-779-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790394221564982-779-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:49:23.261557 22283 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:49:23.262923 22283 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:49:23.275249 22288 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:49:23.275373 22289 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:49:23.275521 22291 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:49:23.276544 22283 server_base.cc:1034] running on GCE node
I20260926 03:49:23.277391 22283 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:49:23.279001 22283 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:49:23.280251 22283 hybrid_clock.cc:648] HybridClock initialized: now 1790394563280218 us; error 52 us; skew 500 ppm
I20260926 03:49:23.280627 22283 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:49:23 dist-test-slave-n2th krb5kdc[22270](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394563, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:49:23.283860 22283 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 03:49:23.285321 22283 webserver.cc:460] Webserver started at http://127.0.0.1:33677/ using document root <none> and password file <none>
I20260926 03:49:23.286002 22283 fs_manager.cc:359] Metadata directory not provided
I20260926 03:49:23.286113 22283 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:49:23.286371 22283 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:49:23.289494 22283 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790394221564982-779-0/minicluster-data/master-0/data/instance:
uuid: "e6f37cf8d5da44beaa19f9154a7f9673"
format_stamp: "Formatted at 2026-09-26 03:49:23 on dist-test-slave-n2th"
server_key: "448be39a4eef04b03dfe9c103bbbfd1f"
server_key_iv: "09ccea9fc0e8986ae13505ce24dfa7f0"
server_key_version: "encryptionkey@0"
I20260926 03:49:23.290288 22283 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790394221564982-779-0/minicluster-data/master-0/wal/instance:
uuid: "e6f37cf8d5da44beaa19f9154a7f9673"
format_stamp: "Formatted at 2026-09-26 03:49:23 on dist-test-slave-n2th"
server_key: "448be39a4eef04b03dfe9c103bbbfd1f"
server_key_iv: "09ccea9fc0e8986ae13505ce24dfa7f0"
server_key_version: "encryptionkey@0"
I20260926 03:49:23.295642 22283 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260926 03:49:23.299341 22298 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:49:23.300817 22283 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260926 03:49:23.301015 22283 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790394221564982-779-0/minicluster-data/master-0/data,/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790394221564982-779-0/minicluster-data/master-0/wal
uuid: "e6f37cf8d5da44beaa19f9154a7f9673"
format_stamp: "Formatted at 2026-09-26 03:49:23 on dist-test-slave-n2th"
server_key: "448be39a4eef04b03dfe9c103bbbfd1f"
server_key_iv: "09ccea9fc0e8986ae13505ce24dfa7f0"
server_key_version: "encryptionkey@0"
I20260926 03:49:23.301237 22283 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790394221564982-779-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790394221564982-779-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790394221564982-779-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:49:23.332746 22283 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:49:23.337252 22283 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:49:23.337571 22283 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:49:23.357111 22283 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46753
I20260926 03:49:23.357146 22349 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46753 every 8 connection(s)
I20260926 03:49:23.358543 22283 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790394221564982-779-0/minicluster-data/master-0/data/info.pb
I20260926 03:49:23.362466 22350 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:49:23.365236 779 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu as pid 22283
I20260926 03:49:23.365420 779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790394221564982-779-0/minicluster-data/master-0/wal/instance
I20260926 03:49:23.365896 779 external_mini_cluster.cc:1428] Setting key 6ea1c9b064c52e9a17d4b63a1191d735
I20260926 03:49:23.376060 22350 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e6f37cf8d5da44beaa19f9154a7f9673: Bootstrap starting.
I20260926 03:49:23.380362 22350 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e6f37cf8d5da44beaa19f9154a7f9673: Neither blocks nor log segments found. Creating new log.
I20260926 03:49:23.381688 22353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:49:23.367930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.192:34289 (local address 127.0.0.1:46753)
0926 03:49:23.368722 (+ 792us) server_negotiation.cc:206] Beginning negotiation
0926 03:49:23.368737 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 03:49:23.368787 (+ 50us) server_negotiation.cc:407] Connection header received
0926 03:49:23.371172 (+ 2385us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:49:23.371207 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:49:23.371778 (+ 571us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:49:23.372265 (+ 487us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:49:23.372812 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:49:23.374925 (+ 2113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:49:23.376370 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:49:23.377363 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:49:23.377538 (+ 175us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:49:23.378847 (+ 1309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:49:23.378859 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:49:23.378876 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:49:23.379580 (+ 704us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:49:23.379682 (+ 102us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:49:23.379714 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:49:23.379864 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0926 03:49:23.380124 (+ 260us) server_negotiation.cc:299] Negotiation successful
0926 03:49:23.380304 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":551,"thread_start_us":319,"threads_started":1}
I20260926 03:49:23.381754 22350 log.cc:824] T 00000000000000000000000000000000 P e6f37cf8d5da44beaa19f9154a7f9673: Log is configured to *not* fsync() on all Append() calls
I20260926 03:49:23.385260 22350 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e6f37cf8d5da44beaa19f9154a7f9673: No bootstrap required, opened a new log
I20260926 03:49:23.386243 779 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 03:49:23.394049 22350 raft_consensus.cc:348] T 00000000000000000000000000000000 P e6f37cf8d5da44beaa19f9154a7f9673 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6f37cf8d5da44beaa19f9154a7f9673" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46753 } }
I20260926 03:49:23.394485 22350 raft_consensus.cc:374] T 00000000000000000000000000000000 P e6f37cf8d5da44beaa19f9154a7f9673 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:49:23.394603 22350 raft_consensus.cc:729] T 00000000000000000000000000000000 P e6f37cf8d5da44beaa19f9154a7f9673 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6f37cf8d5da44beaa19f9154a7f9673, State: Initialized, Role: FOLLOWER
I20260926 03:49:23.395300 22350 consensus_queue.cc:260] T 00000000000000000000000000000000 P e6f37cf8d5da44beaa19f9154a7f9673 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6f37cf8d5da44beaa19f9154a7f9673" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46753 } }
I20260926 03:49:23.395615 22350 raft_consensus.cc:388] T 00000000000000000000000000000000 P e6f37cf8d5da44beaa19f9154a7f9673 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:49:23.395769 22350 raft_consensus.cc:482] T 00000000000000000000000000000000 P e6f37cf8d5da44beaa19f9154a7f9673 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:49:23.395952 22350 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e6f37cf8d5da44beaa19f9154a7f9673 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:49:23.398028 22353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:49:23.390806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.192:38141 (local address 127.0.0.1:46753)
0926 03:49:23.391074 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0926 03:49:23.391084 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 03:49:23.391612 (+ 528us) server_negotiation.cc:407] Connection header received
0926 03:49:23.391755 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:49:23.391760 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:49:23.391844 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:49:23.391955 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:49:23.392452 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:49:23.393596 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:49:23.395245 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:49:23.395918 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:49:23.396064 (+ 146us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:49:23.397212 (+ 1148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:49:23.397218 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:49:23.397224 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:49:23.397245 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:49:23.397317 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:49:23.397337 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:49:23.397451 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0926 03:49:23.397632 (+ 181us) server_negotiation.cc:299] Negotiation successful
0926 03:49:23.397698 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 03:49:23.399140 22350 raft_consensus.cc:504] T 00000000000000000000000000000000 P e6f37cf8d5da44beaa19f9154a7f9673 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6f37cf8d5da44beaa19f9154a7f9673" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46753 } }
I20260926 03:49:23.399690 22350 leader_election.cc:302] T 00000000000000000000000000000000 P e6f37cf8d5da44beaa19f9154a7f9673 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e6f37cf8d5da44beaa19f9154a7f9673; no voters:
I20260926 03:49:23.400270 22350 leader_election.cc:288] T 00000000000000000000000000000000 P e6f37cf8d5da44beaa19f9154a7f9673 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:49:23.400422 22360 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e6f37cf8d5da44beaa19f9154a7f9673 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:49:23.401227 22360 raft_consensus.cc:686] T 00000000000000000000000000000000 P e6f37cf8d5da44beaa19f9154a7f9673 [term 1 LEADER]: Becoming Leader. State: Replica: e6f37cf8d5da44beaa19f9154a7f9673, State: Running, Role: LEADER
I20260926 03:49:23.401762 22360 consensus_queue.cc:237] T 00000000000000000000000000000000 P e6f37cf8d5da44beaa19f9154a7f9673 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6f37cf8d5da44beaa19f9154a7f9673" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46753 } }
I20260926 03:49:23.402437 22350 sys_catalog.cc:564] T 00000000000000000000000000000000 P e6f37cf8d5da44beaa19f9154a7f9673 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:49:23.405130 22362 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6f37cf8d5da44beaa19f9154a7f9673 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e6f37cf8d5da44beaa19f9154a7f9673. Latest consensus state: current_term: 1 leader_uuid: "e6f37cf8d5da44beaa19f9154a7f9673" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6f37cf8d5da44beaa19f9154a7f9673" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46753 } } }
I20260926 03:49:23.405125 22361 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6f37cf8d5da44beaa19f9154a7f9673 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e6f37cf8d5da44beaa19f9154a7f9673" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6f37cf8d5da44beaa19f9154a7f9673" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46753 } } }
I20260926 03:49:23.405542 22362 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6f37cf8d5da44beaa19f9154a7f9673 [sys.catalog]: This master's current role is: LEADER
I20260926 03:49:23.405587 22361 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6f37cf8d5da44beaa19f9154a7f9673 [sys.catalog]: This master's current role is: LEADER
I20260926 03:49:23.411461 22367 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:49:23.417068 22367 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:49:23.426308 22367 catalog_manager.cc:1347] Generated new cluster ID: d0699ac0ca2742f0a93d6286370ef852
I20260926 03:49:23.426445 22367 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:49:23.442430 22367 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:49:23.443833 22367 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:49:23.450188 22367 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e6f37cf8d5da44beaa19f9154a7f9673: Generated new TSK 0
I20260926 03:49:23.451275 22367 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 03:49:23.467937 779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu pid 22283
W20260926 03:49:23.469494 22345 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.192:34289
W20260926 03:49:23.470367 779 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu pid 22283: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 03:49:23.470508 779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu with pid 22283
W20260926 03:49:23.487695 22268 connection.cc:535] client connection to 127.0.0.1:46753 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46753): Connection reset by peer (error 104)
2026-09-26T03:49:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1967 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790394221564982-779-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:49:23 dist-test-slave-n2th krb5kdc[22381](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:49:23 dist-test-slave-n2th krb5kdc[22381](info): set up 2 sockets
Sep 26 03:49:23 dist-test-slave-n2th krb5kdc[22381](info): commencing operation
krb5kdc: starting...
W20260926 03:49:24.599301 779 mini_kdc.cc:121] Time spent starting KDC: real 1.092s user 0.008s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790394221564982-779-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790394221564982-779-0/krb5kdc/test-user.keytab.
Sep 26 03:49:24 dist-test-slave-n2th krb5kdc[22381](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394564, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:49:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:49:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790394221564982-779-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790394221564982-779-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 03:49:24.869925 779 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
/tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790394221564982-779-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790394221564982-779-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790394221564982-779-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790394221564982-779-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36443
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790394221564982-779-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36443
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790394221564982-779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790394221564982-779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790394221564982-779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:49:25.212667 22394 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:49:25.212968 22394 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:49:25.213150 22394 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:49:25.220750 22394 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:49:25.220880 22394 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:49:25.221010 22394 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:49:25.221115 22394 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:49:25.221189 22394 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:49:25.231576 22394 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790394221564982-779-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790394221564982-779-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36443
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790394221564982-779-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36443
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790394221564982-779-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790394221564982-779-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:49:25.232980 22394 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:49:25.234341 22394 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:49:25.246817 22399 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:49:25.246824 22400 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:49:25.247014 22402 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:49:25.247756 22394 server_base.cc:1034] running on GCE node
I20260926 03:49:25.248631 22394 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:49:25.250200 22394 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:49:25.251446 22394 hybrid_clock.cc:648] HybridClock initialized: now 1790394565251400 us; error 44 us; skew 500 ppm
I20260926 03:49:25.251820 22394 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:49:25 dist-test-slave-n2th krb5kdc[22381](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394565, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:49:25.255155 22394 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 03:49:25.256721 22394 webserver.cc:460] Webserver started at http://127.0.0.1:34971/ using document root <none> and password file <none>
I20260926 03:49:25.257434 22394 fs_manager.cc:359] Metadata directory not provided
I20260926 03:49:25.257580 22394 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:49:25.257905 22394 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:49:25.261072 22394 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790394221564982-779-0/minicluster-data/master-0/data/instance:
uuid: "cfb0d4ec7b1541d587b9a937fd51d8db"
format_stamp: "Formatted at 2026-09-26 03:49:25 on dist-test-slave-n2th"
server_key: "878c7c376b2c13d063950347a7398b54"
server_key_iv: "f83431513f351a8a8c9628cb33276db3"
server_key_version: "encryptionkey@0"
I20260926 03:49:25.261941 22394 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790394221564982-779-0/minicluster-data/master-0/wal/instance:
uuid: "cfb0d4ec7b1541d587b9a937fd51d8db"
format_stamp: "Formatted at 2026-09-26 03:49:25 on dist-test-slave-n2th"
server_key: "878c7c376b2c13d063950347a7398b54"
server_key_iv: "f83431513f351a8a8c9628cb33276db3"
server_key_version: "encryptionkey@0"
I20260926 03:49:25.267555 22394 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260926 03:49:25.271282 22409 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:49:25.272667 22394 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260926 03:49:25.272879 22394 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790394221564982-779-0/minicluster-data/master-0/data,/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790394221564982-779-0/minicluster-data/master-0/wal
uuid: "cfb0d4ec7b1541d587b9a937fd51d8db"
format_stamp: "Formatted at 2026-09-26 03:49:25 on dist-test-slave-n2th"
server_key: "878c7c376b2c13d063950347a7398b54"
server_key_iv: "f83431513f351a8a8c9628cb33276db3"
server_key_version: "encryptionkey@0"
I20260926 03:49:25.273102 22394 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790394221564982-779-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790394221564982-779-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790394221564982-779-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:49:25.279111 22394 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:49:25.283703 22394 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:49:25.284034 22394 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:49:25.303726 22394 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36443
I20260926 03:49:25.303771 22460 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36443 every 8 connection(s)
I20260926 03:49:25.305218 22394 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790394221564982-779-0/minicluster-data/master-0/data/info.pb
I20260926 03:49:25.305894 779 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu as pid 22394
I20260926 03:49:25.306110 779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790394221564982-779-0/minicluster-data/master-0/wal/instance
I20260926 03:49:25.306692 779 external_mini_cluster.cc:1428] Setting key ada6561d410639fa49bf296d8d13a17e
I20260926 03:49:25.309995 22461 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:49:25.316013 22464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:49:25.308735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.192:49901 (local address 127.0.0.1:36443)
0926 03:49:25.309560 (+ 825us) server_negotiation.cc:206] Beginning negotiation
0926 03:49:25.309578 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0926 03:49:25.309631 (+ 53us) server_negotiation.cc:407] Connection header received
0926 03:49:25.311970 (+ 2339us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:49:25.312000 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:49:25.312434 (+ 434us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:49:25.312906 (+ 472us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:49:25.313955 (+ 1049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:49:25.313962 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:49:25.313980 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:49:25.314004 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:49:25.314089 (+ 85us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:49:25.314117 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:49:25.314338 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0926 03:49:25.314500 (+ 162us) server_negotiation.cc:299] Negotiation successful
0926 03:49:25.314705 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":549,"thread_start_us":318,"threads_started":1}
I20260926 03:49:25.320068 779 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 03:49:25.323527 22461 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cfb0d4ec7b1541d587b9a937fd51d8db: Bootstrap starting.
I20260926 03:49:25.327292 22464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:49:25.324606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.192:42219 (local address 127.0.0.1:36443)
0926 03:49:25.324786 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0926 03:49:25.324794 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:49:25.325175 (+ 381us) server_negotiation.cc:407] Connection header received
0926 03:49:25.325318 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:49:25.325324 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:49:25.325399 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:49:25.325530 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:49:25.326490 (+ 960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:49:25.326496 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:49:25.326500 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:49:25.326519 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:49:25.326564 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:49:25.326584 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:49:25.326697 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0926 03:49:25.326883 (+ 186us) server_negotiation.cc:299] Negotiation successful
0926 03:49:25.326945 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 03:49:25.327296 22461 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cfb0d4ec7b1541d587b9a937fd51d8db: Neither blocks nor log segments found. Creating new log.
I20260926 03:49:25.328444 22461 log.cc:824] T 00000000000000000000000000000000 P cfb0d4ec7b1541d587b9a937fd51d8db: Log is configured to *not* fsync() on all Append() calls
I20260926 03:49:25.331480 22461 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cfb0d4ec7b1541d587b9a937fd51d8db: No bootstrap required, opened a new log
I20260926 03:49:25.339500 22461 raft_consensus.cc:348] T 00000000000000000000000000000000 P cfb0d4ec7b1541d587b9a937fd51d8db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfb0d4ec7b1541d587b9a937fd51d8db" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36443 } }
I20260926 03:49:25.340049 22461 raft_consensus.cc:374] T 00000000000000000000000000000000 P cfb0d4ec7b1541d587b9a937fd51d8db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:49:25.340185 22461 raft_consensus.cc:729] T 00000000000000000000000000000000 P cfb0d4ec7b1541d587b9a937fd51d8db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfb0d4ec7b1541d587b9a937fd51d8db, State: Initialized, Role: FOLLOWER
I20260926 03:49:25.340997 22461 consensus_queue.cc:260] T 00000000000000000000000000000000 P cfb0d4ec7b1541d587b9a937fd51d8db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfb0d4ec7b1541d587b9a937fd51d8db" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36443 } }
I20260926 03:49:25.341274 22461 raft_consensus.cc:388] T 00000000000000000000000000000000 P cfb0d4ec7b1541d587b9a937fd51d8db [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:49:25.341380 22461 raft_consensus.cc:482] T 00000000000000000000000000000000 P cfb0d4ec7b1541d587b9a937fd51d8db [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:49:25.341511 22461 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cfb0d4ec7b1541d587b9a937fd51d8db [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:49:25.343747 22461 raft_consensus.cc:504] T 00000000000000000000000000000000 P cfb0d4ec7b1541d587b9a937fd51d8db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfb0d4ec7b1541d587b9a937fd51d8db" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36443 } }
I20260926 03:49:25.344218 22461 leader_election.cc:302] T 00000000000000000000000000000000 P cfb0d4ec7b1541d587b9a937fd51d8db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cfb0d4ec7b1541d587b9a937fd51d8db; no voters:
I20260926 03:49:25.344863 22461 leader_election.cc:288] T 00000000000000000000000000000000 P cfb0d4ec7b1541d587b9a937fd51d8db [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:49:25.345031 22471 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cfb0d4ec7b1541d587b9a937fd51d8db [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:49:25.345801 22471 raft_consensus.cc:686] T 00000000000000000000000000000000 P cfb0d4ec7b1541d587b9a937fd51d8db [term 1 LEADER]: Becoming Leader. State: Replica: cfb0d4ec7b1541d587b9a937fd51d8db, State: Running, Role: LEADER
I20260926 03:49:25.346347 22471 consensus_queue.cc:237] T 00000000000000000000000000000000 P cfb0d4ec7b1541d587b9a937fd51d8db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfb0d4ec7b1541d587b9a937fd51d8db" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36443 } }
I20260926 03:49:25.347180 22461 sys_catalog.cc:564] T 00000000000000000000000000000000 P cfb0d4ec7b1541d587b9a937fd51d8db [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:49:25.349355 22473 sys_catalog.cc:455] T 00000000000000000000000000000000 P cfb0d4ec7b1541d587b9a937fd51d8db [sys.catalog]: SysCatalogTable state changed. Reason: New leader cfb0d4ec7b1541d587b9a937fd51d8db. Latest consensus state: current_term: 1 leader_uuid: "cfb0d4ec7b1541d587b9a937fd51d8db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfb0d4ec7b1541d587b9a937fd51d8db" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36443 } } }
I20260926 03:49:25.350023 22473 sys_catalog.cc:458] T 00000000000000000000000000000000 P cfb0d4ec7b1541d587b9a937fd51d8db [sys.catalog]: This master's current role is: LEADER
I20260926 03:49:25.350042 22472 sys_catalog.cc:455] T 00000000000000000000000000000000 P cfb0d4ec7b1541d587b9a937fd51d8db [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cfb0d4ec7b1541d587b9a937fd51d8db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfb0d4ec7b1541d587b9a937fd51d8db" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36443 } } }
I20260926 03:49:25.350558 22472 sys_catalog.cc:458] T 00000000000000000000000000000000 P cfb0d4ec7b1541d587b9a937fd51d8db [sys.catalog]: This master's current role is: LEADER
I20260926 03:49:25.354036 22480 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:49:25.359645 22480 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:49:25.369392 22480 catalog_manager.cc:1347] Generated new cluster ID: 7e537c2f8a684b37b3e45aafa4d27a96
I20260926 03:49:25.369539 22480 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:49:25.384886 22480 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:49:25.385305 22480 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:49:25.404577 22480 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cfb0d4ec7b1541d587b9a937fd51d8db: Generated new TSK 0
I20260926 03:49:25.405652 22480 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 03:49:25.423050 779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu pid 22394
W20260926 03:49:25.424291 22456 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.192:49901
W20260926 03:49:25.425057 779 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu pid 22394: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 03:49:25.425220 779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu with pid 22394
2026-09-26T03:49:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1956 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790394221564982-779-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:49:25 dist-test-slave-n2th krb5kdc[22492](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:49:25 dist-test-slave-n2th krb5kdc[22492](info): set up 2 sockets
Sep 26 03:49:25 dist-test-slave-n2th krb5kdc[22492](info): commencing operation
krb5kdc: starting...
W20260926 03:49:26.559356 779 mini_kdc.cc:121] Time spent starting KDC: real 1.095s user 0.000s sys 0.034s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790394221564982-779-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790394221564982-779-0/krb5kdc/test-user.keytab.
Sep 26 03:49:26 dist-test-slave-n2th krb5kdc[22492](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394566, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:49:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:49:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.194.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.194.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.194.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790394221564982-779-0/krb5kdc/kudu_127.0.194.254.keytab.
Entry for principal kudu/127.0.194.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790394221564982-779-0/krb5kdc/kudu_127.0.194.254.keytab.
I20260926 03:49:26.846917 779 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
/tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790394221564982-779-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790394221564982-779-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790394221564982-779-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790394221564982-779-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.194.254:42001
--webserver_interface=127.0.194.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790394221564982-779-0/krb5kdc/kudu_127.0.194.254.keytab
--principal=kudu/127.0.194.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.194.212:44393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.194.254:42001
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790394221564982-779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790394221564982-779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790394221564982-779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:49:27.212456 22505 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:49:27.212838 22505 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:49:27.213016 22505 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:49:27.220430 22505 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:49:27.220613 22505 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:49:27.220746 22505 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:49:27.220901 22505 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:49:27.220988 22505 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:49:27.231710 22505 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.194.212:44393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790394221564982-779-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790394221564982-779-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.194.254:42001
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790394221564982-779-0/krb5kdc/kudu_127.0.194.254.keytab
--principal=kudu/127.0.194.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.194.254:42001
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790394221564982-779-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.194.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790394221564982-779-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:49:27.233095 22505 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:49:27.234470 22505 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:49:27.247414 22510 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:49:27.247381 22513 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:49:27.247470 22511 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:49:27.248382 22505 server_base.cc:1034] running on GCE node
I20260926 03:49:27.249248 22505 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:49:27.250929 22505 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:49:27.252172 22505 hybrid_clock.cc:648] HybridClock initialized: now 1790394567252110 us; error 66 us; skew 500 ppm
I20260926 03:49:27.252549 22505 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:49:27 dist-test-slave-n2th krb5kdc[22492](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394567, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:49:27.255795 22505 init.cc:377] Logged in from keytab as kudu/127.0.194.254@KRBTEST.COM (short username kudu)
I20260926 03:49:27.257457 22505 webserver.cc:460] Webserver started at http://127.0.194.254:37643/ using document root <none> and password file <none>
I20260926 03:49:27.258213 22505 fs_manager.cc:359] Metadata directory not provided
I20260926 03:49:27.258365 22505 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:49:27.258684 22505 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:49:27.261819 22505 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790394221564982-779-0/minicluster-data/master-0/data/instance:
uuid: "e0baa4fe27b24e70ad194485eba59da9"
format_stamp: "Formatted at 2026-09-26 03:49:27 on dist-test-slave-n2th"
server_key: "59b0471c6ff572140f7319bf9320fe64"
server_key_iv: "81351dca5678c826e315f4d9cfffcbb2"
server_key_version: "encryptionkey@0"
I20260926 03:49:27.262627 22505 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790394221564982-779-0/minicluster-data/master-0/wal/instance:
uuid: "e0baa4fe27b24e70ad194485eba59da9"
format_stamp: "Formatted at 2026-09-26 03:49:27 on dist-test-slave-n2th"
server_key: "59b0471c6ff572140f7319bf9320fe64"
server_key_iv: "81351dca5678c826e315f4d9cfffcbb2"
server_key_version: "encryptionkey@0"
I20260926 03:49:27.268399 22505 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260926 03:49:27.272275 22520 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:49:27.273784 22505 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 03:49:27.274049 22505 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790394221564982-779-0/minicluster-data/master-0/data,/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790394221564982-779-0/minicluster-data/master-0/wal
uuid: "e0baa4fe27b24e70ad194485eba59da9"
format_stamp: "Formatted at 2026-09-26 03:49:27 on dist-test-slave-n2th"
server_key: "59b0471c6ff572140f7319bf9320fe64"
server_key_iv: "81351dca5678c826e315f4d9cfffcbb2"
server_key_version: "encryptionkey@0"
I20260926 03:49:27.274283 22505 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790394221564982-779-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790394221564982-779-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790394221564982-779-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:49:27.294679 22505 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:49:27.299098 22505 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:49:27.299407 22505 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:49:27.319016 22505 rpc_server.cc:307] RPC server started. Bound to: 127.0.194.254:42001
I20260926 03:49:27.319018 22571 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.194.254:42001 every 8 connection(s)
I20260926 03:49:27.320566 22505 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790394221564982-779-0/minicluster-data/master-0/data/info.pb
I20260926 03:49:27.324335 779 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu as pid 22505
I20260926 03:49:27.324525 779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790394221564982-779-0/minicluster-data/master-0/wal/instance
I20260926 03:49:27.324980 779 external_mini_cluster.cc:1428] Setting key 739a6d3645df583e25593395b90ad44e
I20260926 03:49:27.325206 22572 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:49:27.338599 22572 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0baa4fe27b24e70ad194485eba59da9: Bootstrap starting.
Sep 26 03:49:27 dist-test-slave-n2th krb5kdc[22492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394566, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.194.254@KRBTEST.COM
I20260926 03:49:27.342725 22572 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e0baa4fe27b24e70ad194485eba59da9: Neither blocks nor log segments found. Creating new log.
I20260926 03:49:27.344012 22572 log.cc:824] T 00000000000000000000000000000000 P e0baa4fe27b24e70ad194485eba59da9: Log is configured to *not* fsync() on all Append() calls
I20260926 03:49:27.347903 22572 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0baa4fe27b24e70ad194485eba59da9: No bootstrap required, opened a new log
I20260926 03:49:27.350153 22575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:49:27.326862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52393 (local address 127.0.194.254:42001)
0926 03:49:27.327680 (+ 818us) server_negotiation.cc:206] Beginning negotiation
0926 03:49:27.327694 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 03:49:27.327735 (+ 41us) server_negotiation.cc:407] Connection header received
0926 03:49:27.330161 (+ 2426us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:49:27.330186 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:49:27.330665 (+ 479us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:49:27.331194 (+ 529us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:49:27.332689 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:49:27.333996 (+ 1307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:49:27.335397 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:49:27.336168 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:49:27.336396 (+ 228us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:49:27.341374 (+ 4978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:49:27.341384 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:49:27.341401 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:49:27.341451 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:49:27.345304 (+ 3853us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:49:27.346528 (+ 1224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:49:27.346535 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:49:27.346542 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:49:27.346716 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:49:27.347332 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:49:27.347336 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:49:27.347338 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:49:27.347894 (+ 556us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:49:27.348210 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0926 03:49:27.348705 (+ 495us) server_negotiation.cc:299] Negotiation successful
0926 03:49:27.349017 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":604,"thread_start_us":367,"threads_started":1}
I20260926 03:49:27.354502 779 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 03:49:27.357707 22572 raft_consensus.cc:348] T 00000000000000000000000000000000 P e0baa4fe27b24e70ad194485eba59da9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0baa4fe27b24e70ad194485eba59da9" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 42001 } }
I20260926 03:49:27.358314 22572 raft_consensus.cc:374] T 00000000000000000000000000000000 P e0baa4fe27b24e70ad194485eba59da9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:49:27.358438 22572 raft_consensus.cc:729] T 00000000000000000000000000000000 P e0baa4fe27b24e70ad194485eba59da9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0baa4fe27b24e70ad194485eba59da9, State: Initialized, Role: FOLLOWER
I20260926 03:49:27.359293 22572 consensus_queue.cc:260] T 00000000000000000000000000000000 P e0baa4fe27b24e70ad194485eba59da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0baa4fe27b24e70ad194485eba59da9" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 42001 } }
I20260926 03:49:27.359591 22572 raft_consensus.cc:388] T 00000000000000000000000000000000 P e0baa4fe27b24e70ad194485eba59da9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:49:27.359750 22572 raft_consensus.cc:482] T 00000000000000000000000000000000 P e0baa4fe27b24e70ad194485eba59da9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:49:27.359972 22572 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e0baa4fe27b24e70ad194485eba59da9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:49:27.362543 22572 raft_consensus.cc:504] T 00000000000000000000000000000000 P e0baa4fe27b24e70ad194485eba59da9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0baa4fe27b24e70ad194485eba59da9" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 42001 } }
I20260926 03:49:27.363183 22572 leader_election.cc:302] T 00000000000000000000000000000000 P e0baa4fe27b24e70ad194485eba59da9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e0baa4fe27b24e70ad194485eba59da9; no voters:
I20260926 03:49:27.363955 22572 leader_election.cc:288] T 00000000000000000000000000000000 P e0baa4fe27b24e70ad194485eba59da9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:49:27.364064 22582 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e0baa4fe27b24e70ad194485eba59da9 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:49:27.366823 22572 sys_catalog.cc:564] T 00000000000000000000000000000000 P e0baa4fe27b24e70ad194485eba59da9 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:49:27.370926 22582 raft_consensus.cc:686] T 00000000000000000000000000000000 P e0baa4fe27b24e70ad194485eba59da9 [term 1 LEADER]: Becoming Leader. State: Replica: e0baa4fe27b24e70ad194485eba59da9, State: Running, Role: LEADER
I20260926 03:49:27.371865 22582 consensus_queue.cc:237] T 00000000000000000000000000000000 P e0baa4fe27b24e70ad194485eba59da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0baa4fe27b24e70ad194485eba59da9" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 42001 } }
I20260926 03:49:27.375936 22575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:49:27.360224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56285 (local address 127.0.194.254:42001)
0926 03:49:27.360419 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0926 03:49:27.360427 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:49:27.360830 (+ 403us) server_negotiation.cc:407] Connection header received
0926 03:49:27.361073 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:49:27.361079 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:49:27.361162 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:49:27.361279 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:49:27.362801 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:49:27.363673 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:49:27.365548 (+ 1875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:49:27.366012 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:49:27.366169 (+ 157us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:49:27.369164 (+ 2995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:49:27.369170 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:49:27.369173 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:49:27.369252 (+ 79us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:49:27.372686 (+ 3434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:49:27.373640 (+ 954us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:49:27.373646 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:49:27.373651 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:49:27.373800 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:49:27.374300 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:49:27.374307 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:49:27.374315 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:49:27.374643 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:49:27.374982 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0926 03:49:27.375383 (+ 401us) server_negotiation.cc:299] Negotiation successful
0926 03:49:27.375575 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 03:49:27.382102 22583 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0baa4fe27b24e70ad194485eba59da9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e0baa4fe27b24e70ad194485eba59da9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0baa4fe27b24e70ad194485eba59da9" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 42001 } } }
I20260926 03:49:27.382067 22584 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0baa4fe27b24e70ad194485eba59da9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e0baa4fe27b24e70ad194485eba59da9. Latest consensus state: current_term: 1 leader_uuid: "e0baa4fe27b24e70ad194485eba59da9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0baa4fe27b24e70ad194485eba59da9" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 42001 } } }
I20260926 03:49:27.382582 22583 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0baa4fe27b24e70ad194485eba59da9 [sys.catalog]: This master's current role is: LEADER
I20260926 03:49:27.382783 22584 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0baa4fe27b24e70ad194485eba59da9 [sys.catalog]: This master's current role is: LEADER
I20260926 03:49:27.385500 22592 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:49:27.391078 22592 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:49:27.399749 22592 catalog_manager.cc:1347] Generated new cluster ID: a3b6e9d82a3143d295765f88d491adf2
I20260926 03:49:27.399884 22592 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:49:27.420300 22592 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:49:27.422251 22592 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:49:27.431547 22592 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e0baa4fe27b24e70ad194485eba59da9: Generated new TSK 0
I20260926 03:49:27.432479 22592 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 03:49:27.444700 779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu pid 22505
I20260926 03:49:27.445935 22567 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:49:27.538827 779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu with pid 22505
W20260926 03:49:27.556322 22490 connection.cc:535] client connection to 127.0.194.254:42001 recv error: Network error: failed to read from TLS socket (remote: 127.0.194.254:42001): Connection reset by peer (error 104)
2026-09-26T03:49:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (2111 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790394221564982-779-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:49:27 dist-test-slave-n2th krb5kdc[22604](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:49:27 dist-test-slave-n2th krb5kdc[22604](info): set up 2 sockets
Sep 26 03:49:27 dist-test-slave-n2th krb5kdc[22604](info): commencing operation
krb5kdc: starting...
W20260926 03:49:28.652194 779 mini_kdc.cc:121] Time spent starting KDC: real 1.076s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790394221564982-779-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790394221564982-779-0/krb5kdc/test-user.keytab.
Sep 26 03:49:28 dist-test-slave-n2th krb5kdc[22604](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394568, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:49:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:49:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.194.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.194.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.194.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790394221564982-779-0/krb5kdc/kudu_127.0.194.254.keytab.
Entry for principal kudu/127.0.194.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790394221564982-779-0/krb5kdc/kudu_127.0.194.254.keytab.
I20260926 03:49:28.931378 779 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
/tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790394221564982-779-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790394221564982-779-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790394221564982-779-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790394221564982-779-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.194.254:44793
--webserver_interface=127.0.194.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790394221564982-779-0/krb5kdc/kudu_127.0.194.254.keytab
--principal=kudu/127.0.194.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.194.212:42847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.194.254:44793
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790394221564982-779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790394221564982-779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790394221564982-779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:49:29.268036 22617 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:49:29.268352 22617 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:49:29.268541 22617 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:49:29.276077 22617 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:49:29.276202 22617 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:49:29.276300 22617 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:49:29.276418 22617 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:49:29.276504 22617 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:49:29.286659 22617 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.194.212:42847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790394221564982-779-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790394221564982-779-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.194.254:44793
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790394221564982-779-0/krb5kdc/kudu_127.0.194.254.keytab
--principal=kudu/127.0.194.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.194.254:44793
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790394221564982-779-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.194.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790394221564982-779-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:49:29.288172 22617 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:49:29.289534 22617 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:49:29.301744 22622 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:49:29.301808 22623 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:49:29.301839 22625 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:49:29.302418 22617 server_base.cc:1034] running on GCE node
I20260926 03:49:29.303345 22617 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:49:29.304852 22617 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:49:29.306099 22617 hybrid_clock.cc:648] HybridClock initialized: now 1790394569306061 us; error 36 us; skew 500 ppm
I20260926 03:49:29.306468 22617 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:49:29 dist-test-slave-n2th krb5kdc[22604](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394569, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:49:29.309691 22617 init.cc:377] Logged in from keytab as kudu/127.0.194.254@KRBTEST.COM (short username kudu)
I20260926 03:49:29.311326 22617 webserver.cc:460] Webserver started at http://127.0.194.254:42857/ using document root <none> and password file <none>
I20260926 03:49:29.312083 22617 fs_manager.cc:359] Metadata directory not provided
I20260926 03:49:29.312242 22617 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:49:29.312572 22617 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:49:29.315739 22617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790394221564982-779-0/minicluster-data/master-0/data/instance:
uuid: "675c4fdd5c02418ba3c5f1eeca82f765"
format_stamp: "Formatted at 2026-09-26 03:49:29 on dist-test-slave-n2th"
server_key: "30610dc4510493afee23c1a1a92d11f4"
server_key_iv: "7b748535acae626e995c72994021b99a"
server_key_version: "encryptionkey@0"
I20260926 03:49:29.316557 22617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790394221564982-779-0/minicluster-data/master-0/wal/instance:
uuid: "675c4fdd5c02418ba3c5f1eeca82f765"
format_stamp: "Formatted at 2026-09-26 03:49:29 on dist-test-slave-n2th"
server_key: "30610dc4510493afee23c1a1a92d11f4"
server_key_iv: "7b748535acae626e995c72994021b99a"
server_key_version: "encryptionkey@0"
I20260926 03:49:29.321995 22617 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 03:49:29.325824 22632 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:49:29.327418 22617 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 03:49:29.327649 22617 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790394221564982-779-0/minicluster-data/master-0/data,/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790394221564982-779-0/minicluster-data/master-0/wal
uuid: "675c4fdd5c02418ba3c5f1eeca82f765"
format_stamp: "Formatted at 2026-09-26 03:49:29 on dist-test-slave-n2th"
server_key: "30610dc4510493afee23c1a1a92d11f4"
server_key_iv: "7b748535acae626e995c72994021b99a"
server_key_version: "encryptionkey@0"
I20260926 03:49:29.327879 22617 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790394221564982-779-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790394221564982-779-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790394221564982-779-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:49:29.347762 22617 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:49:29.352047 22617 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:49:29.352381 22617 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:49:29.372138 22617 rpc_server.cc:307] RPC server started. Bound to: 127.0.194.254:44793
I20260926 03:49:29.372133 22683 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.194.254:44793 every 8 connection(s)
I20260926 03:49:29.373713 22617 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790394221564982-779-0/minicluster-data/master-0/data/info.pb
I20260926 03:49:29.377480 779 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu as pid 22617
I20260926 03:49:29.377678 779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790394221564982-779-0/minicluster-data/master-0/wal/instance
I20260926 03:49:29.378021 22684 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:49:29.378175 779 external_mini_cluster.cc:1428] Setting key 1a4b27ee7b2eb985c409eb8b83073bde
I20260926 03:49:29.392764 22684 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 675c4fdd5c02418ba3c5f1eeca82f765: Bootstrap starting.
I20260926 03:49:29.392871 22687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:49:29.380198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48585 (local address 127.0.194.254:44793)
0926 03:49:29.381053 (+ 855us) server_negotiation.cc:206] Beginning negotiation
0926 03:49:29.381064 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 03:49:29.381110 (+ 46us) server_negotiation.cc:407] Connection header received
0926 03:49:29.383204 (+ 2094us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:49:29.383230 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:49:29.383691 (+ 461us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:49:29.384060 (+ 369us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:49:29.384721 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:49:29.386098 (+ 1377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:49:29.387961 (+ 1863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:49:29.388727 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:49:29.388912 (+ 185us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:49:29.390337 (+ 1425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:49:29.390344 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:49:29.390361 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:49:29.391036 (+ 675us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:49:29.391139 (+ 103us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:49:29.391164 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:49:29.391286 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0926 03:49:29.391616 (+ 330us) server_negotiation.cc:299] Negotiation successful
0926 03:49:29.391816 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":623,"thread_start_us":389,"threads_started":1}
I20260926 03:49:29.396759 22684 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 675c4fdd5c02418ba3c5f1eeca82f765: Neither blocks nor log segments found. Creating new log.
I20260926 03:49:29.397379 779 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 03:49:29.397979 22684 log.cc:824] T 00000000000000000000000000000000 P 675c4fdd5c02418ba3c5f1eeca82f765: Log is configured to *not* fsync() on all Append() calls
I20260926 03:49:29.401116 22684 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 675c4fdd5c02418ba3c5f1eeca82f765: No bootstrap required, opened a new log
I20260926 03:49:29.408461 22687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:49:29.401945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48357 (local address 127.0.194.254:44793)
0926 03:49:29.402155 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0926 03:49:29.402165 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 03:49:29.402555 (+ 390us) server_negotiation.cc:407] Connection header received
0926 03:49:29.402817 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:49:29.402820 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:49:29.402883 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:49:29.403009 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:49:29.403534 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:49:29.404352 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:49:29.405754 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:49:29.406245 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:49:29.406412 (+ 167us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:49:29.407682 (+ 1270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:49:29.407687 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:49:29.407690 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:49:29.407711 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:49:29.407749 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:49:29.407764 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:49:29.407903 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0926 03:49:29.408111 (+ 208us) server_negotiation.cc:299] Negotiation successful
0926 03:49:29.408176 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 03:49:29.410043 22684 raft_consensus.cc:348] T 00000000000000000000000000000000 P 675c4fdd5c02418ba3c5f1eeca82f765 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "675c4fdd5c02418ba3c5f1eeca82f765" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 44793 } }
I20260926 03:49:29.410521 22684 raft_consensus.cc:374] T 00000000000000000000000000000000 P 675c4fdd5c02418ba3c5f1eeca82f765 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:49:29.410633 22684 raft_consensus.cc:729] T 00000000000000000000000000000000 P 675c4fdd5c02418ba3c5f1eeca82f765 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 675c4fdd5c02418ba3c5f1eeca82f765, State: Initialized, Role: FOLLOWER
I20260926 03:49:29.411334 22684 consensus_queue.cc:260] T 00000000000000000000000000000000 P 675c4fdd5c02418ba3c5f1eeca82f765 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "675c4fdd5c02418ba3c5f1eeca82f765" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 44793 } }
I20260926 03:49:29.411587 22684 raft_consensus.cc:388] T 00000000000000000000000000000000 P 675c4fdd5c02418ba3c5f1eeca82f765 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:49:29.411741 22684 raft_consensus.cc:482] T 00000000000000000000000000000000 P 675c4fdd5c02418ba3c5f1eeca82f765 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:49:29.411922 22684 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 675c4fdd5c02418ba3c5f1eeca82f765 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:49:29.414238 22684 raft_consensus.cc:504] T 00000000000000000000000000000000 P 675c4fdd5c02418ba3c5f1eeca82f765 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "675c4fdd5c02418ba3c5f1eeca82f765" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 44793 } }
I20260926 03:49:29.414873 22684 leader_election.cc:302] T 00000000000000000000000000000000 P 675c4fdd5c02418ba3c5f1eeca82f765 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 675c4fdd5c02418ba3c5f1eeca82f765; no voters:
I20260926 03:49:29.415517 22684 leader_election.cc:288] T 00000000000000000000000000000000 P 675c4fdd5c02418ba3c5f1eeca82f765 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:49:29.415624 22694 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 675c4fdd5c02418ba3c5f1eeca82f765 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:49:29.416849 22694 raft_consensus.cc:686] T 00000000000000000000000000000000 P 675c4fdd5c02418ba3c5f1eeca82f765 [term 1 LEADER]: Becoming Leader. State: Replica: 675c4fdd5c02418ba3c5f1eeca82f765, State: Running, Role: LEADER
I20260926 03:49:29.417421 22694 consensus_queue.cc:237] T 00000000000000000000000000000000 P 675c4fdd5c02418ba3c5f1eeca82f765 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "675c4fdd5c02418ba3c5f1eeca82f765" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 44793 } }
I20260926 03:49:29.417707 22684 sys_catalog.cc:564] T 00000000000000000000000000000000 P 675c4fdd5c02418ba3c5f1eeca82f765 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:49:29.420781 22696 sys_catalog.cc:455] T 00000000000000000000000000000000 P 675c4fdd5c02418ba3c5f1eeca82f765 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 675c4fdd5c02418ba3c5f1eeca82f765. Latest consensus state: current_term: 1 leader_uuid: "675c4fdd5c02418ba3c5f1eeca82f765" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "675c4fdd5c02418ba3c5f1eeca82f765" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 44793 } } }
I20260926 03:49:29.420928 22695 sys_catalog.cc:455] T 00000000000000000000000000000000 P 675c4fdd5c02418ba3c5f1eeca82f765 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "675c4fdd5c02418ba3c5f1eeca82f765" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "675c4fdd5c02418ba3c5f1eeca82f765" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 44793 } } }
I20260926 03:49:29.421188 22696 sys_catalog.cc:458] T 00000000000000000000000000000000 P 675c4fdd5c02418ba3c5f1eeca82f765 [sys.catalog]: This master's current role is: LEADER
I20260926 03:49:29.421265 22695 sys_catalog.cc:458] T 00000000000000000000000000000000 P 675c4fdd5c02418ba3c5f1eeca82f765 [sys.catalog]: This master's current role is: LEADER
I20260926 03:49:29.422464 22703 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:49:29.428316 22703 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:49:29.438105 22703 catalog_manager.cc:1347] Generated new cluster ID: 7f9034b9f429488aa9aef15938d53150
I20260926 03:49:29.438243 22703 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:49:29.465133 22703 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:49:29.467136 22703 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:49:29.483916 22703 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 675c4fdd5c02418ba3c5f1eeca82f765: Generated new TSK 0
I20260926 03:49:29.485092 22703 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 03:49:29.492057 779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu pid 22617
W20260926 03:49:29.493271 22679 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:48585
W20260926 03:49:29.494035 779 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu pid 22617: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 03:49:29.494194 779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu with pid 22617
W20260926 03:49:29.511795 22602 connection.cc:535] client connection to 127.0.194.254:44793 recv error: Network error: failed to read from TLS socket (remote: 127.0.194.254:44793): Connection reset by peer (error 104)
2026-09-26T03:49:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1955 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790394221564982-779-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:49:29 dist-test-slave-n2th krb5kdc[22715](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:49:29 dist-test-slave-n2th krb5kdc[22715](info): set up 2 sockets
Sep 26 03:49:29 dist-test-slave-n2th krb5kdc[22715](info): commencing operation
krb5kdc: starting...
W20260926 03:49:30.611264 779 mini_kdc.cc:121] Time spent starting KDC: real 1.080s user 0.001s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790394221564982-779-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790394221564982-779-0/krb5kdc/test-user.keytab.
Sep 26 03:49:30 dist-test-slave-n2th krb5kdc[22715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394570, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:49:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:49:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.194.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.194.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.194.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790394221564982-779-0/krb5kdc/kudu_127.0.194.254.keytab.
Entry for principal kudu/127.0.194.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790394221564982-779-0/krb5kdc/kudu_127.0.194.254.keytab.
I20260926 03:49:30.877628 779 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
/tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790394221564982-779-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790394221564982-779-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790394221564982-779-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790394221564982-779-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.194.254:38477
--webserver_interface=127.0.194.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790394221564982-779-0/krb5kdc/kudu_127.0.194.254.keytab
--principal=kudu/127.0.194.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.194.212:37783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.194.254:38477
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790394221564982-779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790394221564982-779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790394221564982-779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:49:31.223559 22728 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:49:31.223869 22728 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:49:31.224025 22728 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:49:31.231961 22728 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:49:31.232155 22728 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:49:31.232272 22728 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:49:31.232416 22728 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:49:31.232471 22728 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:49:31.242810 22728 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.194.212:37783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790394221564982-779-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790394221564982-779-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.194.254:38477
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790394221564982-779-0/krb5kdc/kudu_127.0.194.254.keytab
--principal=kudu/127.0.194.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.194.254:38477
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790394221564982-779-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.194.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790394221564982-779-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:49:31.244194 22728 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:49:31.246044 22728 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:49:31.259081 22733 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:49:31.259182 22734 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:49:31.259130 22736 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:49:31.259797 22728 server_base.cc:1034] running on GCE node
I20260926 03:49:31.260711 22728 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:49:31.262318 22728 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:49:31.263619 22728 hybrid_clock.cc:648] HybridClock initialized: now 1790394571263549 us; error 62 us; skew 500 ppm
I20260926 03:49:31.264042 22728 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:49:31 dist-test-slave-n2th krb5kdc[22715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394571, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:49:31.267791 22728 init.cc:377] Logged in from keytab as kudu/127.0.194.254@KRBTEST.COM (short username kudu)
I20260926 03:49:31.269520 22728 webserver.cc:460] Webserver started at http://127.0.194.254:39205/ using document root <none> and password file <none>
I20260926 03:49:31.270252 22728 fs_manager.cc:359] Metadata directory not provided
I20260926 03:49:31.270363 22728 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:49:31.270617 22728 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:49:31.273957 22728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790394221564982-779-0/minicluster-data/master-0/data/instance:
uuid: "c16ffcd8683e4dafa8c1674b9de7a6ee"
format_stamp: "Formatted at 2026-09-26 03:49:31 on dist-test-slave-n2th"
server_key: "0a118c915bfc31a288f352e935cb32a2"
server_key_iv: "9e6bb7091f335ad43f0a280fdf74661b"
server_key_version: "encryptionkey@0"
I20260926 03:49:31.274834 22728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790394221564982-779-0/minicluster-data/master-0/wal/instance:
uuid: "c16ffcd8683e4dafa8c1674b9de7a6ee"
format_stamp: "Formatted at 2026-09-26 03:49:31 on dist-test-slave-n2th"
server_key: "0a118c915bfc31a288f352e935cb32a2"
server_key_iv: "9e6bb7091f335ad43f0a280fdf74661b"
server_key_version: "encryptionkey@0"
I20260926 03:49:31.280803 22728 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260926 03:49:31.284952 22743 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:49:31.286484 22728 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 03:49:31.286654 22728 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790394221564982-779-0/minicluster-data/master-0/data,/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790394221564982-779-0/minicluster-data/master-0/wal
uuid: "c16ffcd8683e4dafa8c1674b9de7a6ee"
format_stamp: "Formatted at 2026-09-26 03:49:31 on dist-test-slave-n2th"
server_key: "0a118c915bfc31a288f352e935cb32a2"
server_key_iv: "9e6bb7091f335ad43f0a280fdf74661b"
server_key_version: "encryptionkey@0"
I20260926 03:49:31.286895 22728 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790394221564982-779-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790394221564982-779-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790394221564982-779-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:49:31.293099 22728 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:49:31.297312 22728 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:49:31.297577 22728 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:49:31.318257 22728 rpc_server.cc:307] RPC server started. Bound to: 127.0.194.254:38477
I20260926 03:49:31.318277 22794 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.194.254:38477 every 8 connection(s)
I20260926 03:49:31.320279 22728 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790394221564982-779-0/minicluster-data/master-0/data/info.pb
I20260926 03:49:31.324074 779 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu as pid 22728
I20260926 03:49:31.324286 779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790394221564982-779-0/minicluster-data/master-0/wal/instance
I20260926 03:49:31.324858 779 external_mini_cluster.cc:1428] Setting key 203ba6bb71d61b88a2d978c31fe11888
I20260926 03:49:31.325062 22795 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:49:31.334125 22798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:49:31.326895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53037 (local address 127.0.194.254:38477)
0926 03:49:31.327720 (+ 825us) server_negotiation.cc:206] Beginning negotiation
0926 03:49:31.327732 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 03:49:31.327783 (+ 51us) server_negotiation.cc:407] Connection header received
0926 03:49:31.330010 (+ 2227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:49:31.330064 (+ 54us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:49:31.330490 (+ 426us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:49:31.330990 (+ 500us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:49:31.332204 (+ 1214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:49:31.332211 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:49:31.332226 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:49:31.332244 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:49:31.332328 (+ 84us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:49:31.332350 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:49:31.332644 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0926 03:49:31.332839 (+ 195us) server_negotiation.cc:299] Negotiation successful
0926 03:49:31.333081 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":607,"thread_start_us":386,"threads_started":1}
I20260926 03:49:31.338945 779 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 03:49:31.340466 22795 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c16ffcd8683e4dafa8c1674b9de7a6ee: Bootstrap starting.
I20260926 03:49:31.345276 22795 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c16ffcd8683e4dafa8c1674b9de7a6ee: Neither blocks nor log segments found. Creating new log.
I20260926 03:49:31.346721 22795 log.cc:824] T 00000000000000000000000000000000 P c16ffcd8683e4dafa8c1674b9de7a6ee: Log is configured to *not* fsync() on all Append() calls
I20260926 03:49:31.348178 22798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:49:31.344804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53117 (local address 127.0.194.254:38477)
0926 03:49:31.345080 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0926 03:49:31.345088 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:49:31.345599 (+ 511us) server_negotiation.cc:407] Connection header received
0926 03:49:31.345782 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:49:31.345793 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:49:31.345886 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:49:31.346008 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:49:31.347221 (+ 1213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:49:31.347227 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:49:31.347232 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:49:31.347255 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:49:31.347322 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:49:31.347349 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:49:31.347540 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0926 03:49:31.347741 (+ 201us) server_negotiation.cc:299] Negotiation successful
0926 03:49:31.347817 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118}
I20260926 03:49:31.350510 22795 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c16ffcd8683e4dafa8c1674b9de7a6ee: No bootstrap required, opened a new log
I20260926 03:49:31.358342 22795 raft_consensus.cc:348] T 00000000000000000000000000000000 P c16ffcd8683e4dafa8c1674b9de7a6ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c16ffcd8683e4dafa8c1674b9de7a6ee" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 38477 } }
I20260926 03:49:31.358803 22795 raft_consensus.cc:374] T 00000000000000000000000000000000 P c16ffcd8683e4dafa8c1674b9de7a6ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:49:31.358935 22795 raft_consensus.cc:729] T 00000000000000000000000000000000 P c16ffcd8683e4dafa8c1674b9de7a6ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c16ffcd8683e4dafa8c1674b9de7a6ee, State: Initialized, Role: FOLLOWER
I20260926 03:49:31.359597 22795 consensus_queue.cc:260] T 00000000000000000000000000000000 P c16ffcd8683e4dafa8c1674b9de7a6ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c16ffcd8683e4dafa8c1674b9de7a6ee" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 38477 } }
I20260926 03:49:31.359881 22795 raft_consensus.cc:388] T 00000000000000000000000000000000 P c16ffcd8683e4dafa8c1674b9de7a6ee [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:49:31.360059 22795 raft_consensus.cc:482] T 00000000000000000000000000000000 P c16ffcd8683e4dafa8c1674b9de7a6ee [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:49:31.360255 22795 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c16ffcd8683e4dafa8c1674b9de7a6ee [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:49:31.362531 22795 raft_consensus.cc:504] T 00000000000000000000000000000000 P c16ffcd8683e4dafa8c1674b9de7a6ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c16ffcd8683e4dafa8c1674b9de7a6ee" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 38477 } }
I20260926 03:49:31.363098 22795 leader_election.cc:302] T 00000000000000000000000000000000 P c16ffcd8683e4dafa8c1674b9de7a6ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c16ffcd8683e4dafa8c1674b9de7a6ee; no voters:
I20260926 03:49:31.363770 22795 leader_election.cc:288] T 00000000000000000000000000000000 P c16ffcd8683e4dafa8c1674b9de7a6ee [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:49:31.363946 22805 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c16ffcd8683e4dafa8c1674b9de7a6ee [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:49:31.365303 22805 raft_consensus.cc:686] T 00000000000000000000000000000000 P c16ffcd8683e4dafa8c1674b9de7a6ee [term 1 LEADER]: Becoming Leader. State: Replica: c16ffcd8683e4dafa8c1674b9de7a6ee, State: Running, Role: LEADER
I20260926 03:49:31.365909 22805 consensus_queue.cc:237] T 00000000000000000000000000000000 P c16ffcd8683e4dafa8c1674b9de7a6ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c16ffcd8683e4dafa8c1674b9de7a6ee" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 38477 } }
I20260926 03:49:31.366068 22795 sys_catalog.cc:564] T 00000000000000000000000000000000 P c16ffcd8683e4dafa8c1674b9de7a6ee [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:49:31.368986 22806 sys_catalog.cc:455] T 00000000000000000000000000000000 P c16ffcd8683e4dafa8c1674b9de7a6ee [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c16ffcd8683e4dafa8c1674b9de7a6ee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c16ffcd8683e4dafa8c1674b9de7a6ee" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 38477 } } }
I20260926 03:49:31.369491 22806 sys_catalog.cc:458] T 00000000000000000000000000000000 P c16ffcd8683e4dafa8c1674b9de7a6ee [sys.catalog]: This master's current role is: LEADER
I20260926 03:49:31.370349 22814 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:49:31.369019 22807 sys_catalog.cc:455] T 00000000000000000000000000000000 P c16ffcd8683e4dafa8c1674b9de7a6ee [sys.catalog]: SysCatalogTable state changed. Reason: New leader c16ffcd8683e4dafa8c1674b9de7a6ee. Latest consensus state: current_term: 1 leader_uuid: "c16ffcd8683e4dafa8c1674b9de7a6ee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c16ffcd8683e4dafa8c1674b9de7a6ee" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 38477 } } }
I20260926 03:49:31.371799 22807 sys_catalog.cc:458] T 00000000000000000000000000000000 P c16ffcd8683e4dafa8c1674b9de7a6ee [sys.catalog]: This master's current role is: LEADER
I20260926 03:49:31.377578 22814 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:49:31.389806 22814 catalog_manager.cc:1347] Generated new cluster ID: 62042e532e03412480bcc51c917b61f8
I20260926 03:49:31.389945 22814 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:49:31.422577 22814 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:49:31.423210 22814 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:49:31.446228 22814 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c16ffcd8683e4dafa8c1674b9de7a6ee: Generated new TSK 0
I20260926 03:49:31.447255 22814 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 03:49:31.457559 779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu pid 22728
W20260926 03:49:31.458788 22790 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:53037
W20260926 03:49:31.459558 779 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu pid 22728: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 03:49:31.459721 779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu with pid 22728
2026-09-26T03:49:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1964 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790394221564982-779-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:49:31 dist-test-slave-n2th krb5kdc[22826](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:49:31 dist-test-slave-n2th krb5kdc[22826](info): set up 2 sockets
Sep 26 03:49:31 dist-test-slave-n2th krb5kdc[22826](info): commencing operation
krb5kdc: starting...
W20260926 03:49:32.567349 779 mini_kdc.cc:121] Time spent starting KDC: real 1.071s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790394221564982-779-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790394221564982-779-0/krb5kdc/test-user.keytab.
Sep 26 03:49:32 dist-test-slave-n2th krb5kdc[22826](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394572, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:49:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:49:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.194.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.194.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.194.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790394221564982-779-0/krb5kdc/kudu_127.0.194.254.keytab.
Entry for principal kudu/127.0.194.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790394221564982-779-0/krb5kdc/kudu_127.0.194.254.keytab.
I20260926 03:49:32.830582 779 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
/tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790394221564982-779-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790394221564982-779-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790394221564982-779-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790394221564982-779-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.194.254:43611
--webserver_interface=127.0.194.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790394221564982-779-0/krb5kdc/kudu_127.0.194.254.keytab
--principal=kudu/127.0.194.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.194.212:46019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.194.254:43611
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790394221564982-779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790394221564982-779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790394221564982-779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:49:33.176638 22839 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:49:33.176956 22839 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:49:33.177053 22839 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:49:33.184489 22839 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:49:33.184619 22839 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:49:33.184667 22839 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:49:33.184731 22839 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:49:33.184777 22839 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:49:33.194919 22839 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.194.212:46019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790394221564982-779-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790394221564982-779-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.194.254:43611
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790394221564982-779-0/krb5kdc/kudu_127.0.194.254.keytab
--principal=kudu/127.0.194.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.194.254:43611
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790394221564982-779-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.194.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790394221564982-779-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:49:33.196442 22839 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:49:33.197726 22839 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:49:33.209798 22844 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:49:33.209769 22845 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:49:33.209970 22847 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:49:33.210667 22839 server_base.cc:1034] running on GCE node
I20260926 03:49:33.211541 22839 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:49:33.213028 22839 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:49:33.214244 22839 hybrid_clock.cc:648] HybridClock initialized: now 1790394573214198 us; error 40 us; skew 500 ppm
I20260926 03:49:33.214632 22839 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:49:33 dist-test-slave-n2th krb5kdc[22826](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394573, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:49:33.224372 22839 init.cc:377] Logged in from keytab as kudu/127.0.194.254@KRBTEST.COM (short username kudu)
I20260926 03:49:33.226114 22839 webserver.cc:460] Webserver started at http://127.0.194.254:46715/ using document root <none> and password file <none>
I20260926 03:49:33.226982 22839 fs_manager.cc:359] Metadata directory not provided
I20260926 03:49:33.227141 22839 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:49:33.227465 22839 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:49:33.230680 22839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790394221564982-779-0/minicluster-data/master-0/data/instance:
uuid: "d1e086c926ed4853878bb394d16e533d"
format_stamp: "Formatted at 2026-09-26 03:49:33 on dist-test-slave-n2th"
server_key: "3490f38c8b15319001549a42f2b01baa"
server_key_iv: "006e6bb448a0827694be2917f2b994c2"
server_key_version: "encryptionkey@0"
I20260926 03:49:33.231587 22839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790394221564982-779-0/minicluster-data/master-0/wal/instance:
uuid: "d1e086c926ed4853878bb394d16e533d"
format_stamp: "Formatted at 2026-09-26 03:49:33 on dist-test-slave-n2th"
server_key: "3490f38c8b15319001549a42f2b01baa"
server_key_iv: "006e6bb448a0827694be2917f2b994c2"
server_key_version: "encryptionkey@0"
I20260926 03:49:33.237380 22839 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.002s
I20260926 03:49:33.241079 22854 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:49:33.242413 22839 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 03:49:33.242626 22839 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790394221564982-779-0/minicluster-data/master-0/data,/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790394221564982-779-0/minicluster-data/master-0/wal
uuid: "d1e086c926ed4853878bb394d16e533d"
format_stamp: "Formatted at 2026-09-26 03:49:33 on dist-test-slave-n2th"
server_key: "3490f38c8b15319001549a42f2b01baa"
server_key_iv: "006e6bb448a0827694be2917f2b994c2"
server_key_version: "encryptionkey@0"
I20260926 03:49:33.242913 22839 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790394221564982-779-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790394221564982-779-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790394221564982-779-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:49:33.257411 22839 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:49:33.262174 22839 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:49:33.262494 22839 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:49:33.281917 22839 rpc_server.cc:307] RPC server started. Bound to: 127.0.194.254:43611
I20260926 03:49:33.281925 22905 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.194.254:43611 every 8 connection(s)
I20260926 03:49:33.283411 22839 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790394221564982-779-0/minicluster-data/master-0/data/info.pb
I20260926 03:49:33.287349 22906 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:49:33.287883 779 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu as pid 22839
I20260926 03:49:33.288072 779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790394221564982-779-0/minicluster-data/master-0/wal/instance
I20260926 03:49:33.288527 779 external_mini_cluster.cc:1428] Setting key 1ebad9a6a13f1bba2b7eb068d89a3180
I20260926 03:49:33.302428 22906 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1e086c926ed4853878bb394d16e533d: Bootstrap starting.
Sep 26 03:49:33 dist-test-slave-n2th krb5kdc[22826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394572, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.194.254@KRBTEST.COM
I20260926 03:49:33.306226 22906 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d1e086c926ed4853878bb394d16e533d: Neither blocks nor log segments found. Creating new log.
I20260926 03:49:33.307801 22906 log.cc:824] T 00000000000000000000000000000000 P d1e086c926ed4853878bb394d16e533d: Log is configured to *not* fsync() on all Append() calls
I20260926 03:49:33.311473 22906 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1e086c926ed4853878bb394d16e533d: No bootstrap required, opened a new log
I20260926 03:49:33.314417 22909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:49:33.290484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55397 (local address 127.0.194.254:43611)
0926 03:49:33.291361 (+ 877us) server_negotiation.cc:206] Beginning negotiation
0926 03:49:33.291376 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 03:49:33.291437 (+ 61us) server_negotiation.cc:407] Connection header received
0926 03:49:33.293756 (+ 2319us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:49:33.293793 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:49:33.294369 (+ 576us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:49:33.294918 (+ 549us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:49:33.296364 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:49:33.298304 (+ 1940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:49:33.299707 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:49:33.300817 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:49:33.301017 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:49:33.305517 (+ 4500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:49:33.305528 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:49:33.305547 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:49:33.305596 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:49:33.309561 (+ 3965us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:49:33.310359 (+ 798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:49:33.310371 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:49:33.310382 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:49:33.310616 (+ 234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:49:33.311065 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:49:33.311071 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:49:33.311075 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:49:33.311790 (+ 715us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:49:33.312169 (+ 379us) server_negotiation.cc:1035] Waiting for connection context
0926 03:49:33.312483 (+ 314us) server_negotiation.cc:299] Negotiation successful
0926 03:49:33.312962 (+ 479us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":640,"thread_start_us":406,"threads_started":1}
I20260926 03:49:33.318547 779 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 03:49:33.321019 22906 raft_consensus.cc:348] T 00000000000000000000000000000000 P d1e086c926ed4853878bb394d16e533d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1e086c926ed4853878bb394d16e533d" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 43611 } }
I20260926 03:49:33.321540 22906 raft_consensus.cc:374] T 00000000000000000000000000000000 P d1e086c926ed4853878bb394d16e533d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:49:33.321656 22906 raft_consensus.cc:729] T 00000000000000000000000000000000 P d1e086c926ed4853878bb394d16e533d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1e086c926ed4853878bb394d16e533d, State: Initialized, Role: FOLLOWER
I20260926 03:49:33.322345 22906 consensus_queue.cc:260] T 00000000000000000000000000000000 P d1e086c926ed4853878bb394d16e533d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1e086c926ed4853878bb394d16e533d" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 43611 } }
I20260926 03:49:33.322609 22906 raft_consensus.cc:388] T 00000000000000000000000000000000 P d1e086c926ed4853878bb394d16e533d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:49:33.322777 22906 raft_consensus.cc:482] T 00000000000000000000000000000000 P d1e086c926ed4853878bb394d16e533d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:49:33.322965 22906 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d1e086c926ed4853878bb394d16e533d [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:49:33.325577 22906 raft_consensus.cc:504] T 00000000000000000000000000000000 P d1e086c926ed4853878bb394d16e533d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1e086c926ed4853878bb394d16e533d" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 43611 } }
I20260926 03:49:33.326133 22906 leader_election.cc:302] T 00000000000000000000000000000000 P d1e086c926ed4853878bb394d16e533d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d1e086c926ed4853878bb394d16e533d; no voters:
I20260926 03:49:33.326880 22906 leader_election.cc:288] T 00000000000000000000000000000000 P d1e086c926ed4853878bb394d16e533d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:49:33.327031 22916 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d1e086c926ed4853878bb394d16e533d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:49:33.328830 22916 raft_consensus.cc:686] T 00000000000000000000000000000000 P d1e086c926ed4853878bb394d16e533d [term 1 LEADER]: Becoming Leader. State: Replica: d1e086c926ed4853878bb394d16e533d, State: Running, Role: LEADER
I20260926 03:49:33.329558 22906 sys_catalog.cc:564] T 00000000000000000000000000000000 P d1e086c926ed4853878bb394d16e533d [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:49:33.329536 22916 consensus_queue.cc:237] T 00000000000000000000000000000000 P d1e086c926ed4853878bb394d16e533d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1e086c926ed4853878bb394d16e533d" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 43611 } }
I20260926 03:49:33.332798 22917 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1e086c926ed4853878bb394d16e533d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d1e086c926ed4853878bb394d16e533d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1e086c926ed4853878bb394d16e533d" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 43611 } } }
I20260926 03:49:33.333302 22917 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1e086c926ed4853878bb394d16e533d [sys.catalog]: This master's current role is: LEADER
I20260926 03:49:33.342398 22909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:49:33.323417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44277 (local address 127.0.194.254:43611)
0926 03:49:33.323601 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0926 03:49:33.323608 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:49:33.324028 (+ 420us) server_negotiation.cc:407] Connection header received
0926 03:49:33.324241 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:49:33.324246 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:49:33.324327 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:49:33.324442 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:49:33.325959 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:49:33.326922 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:49:33.328605 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:49:33.329273 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:49:33.329427 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:49:33.332445 (+ 3018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:49:33.332453 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:49:33.332461 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:49:33.332505 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:49:33.336895 (+ 4390us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:49:33.338400 (+ 1505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:49:33.338409 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:49:33.338415 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:49:33.338666 (+ 251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:49:33.339346 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:49:33.339354 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:49:33.339360 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:49:33.339916 (+ 556us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:49:33.340251 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0926 03:49:33.341781 (+ 1530us) server_negotiation.cc:299] Negotiation successful
0926 03:49:33.342018 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 03:49:33.342471 22918 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1e086c926ed4853878bb394d16e533d [sys.catalog]: SysCatalogTable state changed. Reason: New leader d1e086c926ed4853878bb394d16e533d. Latest consensus state: current_term: 1 leader_uuid: "d1e086c926ed4853878bb394d16e533d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1e086c926ed4853878bb394d16e533d" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 43611 } } }
I20260926 03:49:33.342859 22918 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1e086c926ed4853878bb394d16e533d [sys.catalog]: This master's current role is: LEADER
I20260926 03:49:33.344308 22922 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:49:33.350320 22922 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:49:33.359712 22922 catalog_manager.cc:1347] Generated new cluster ID: d05880358e3a4307b37acd474388ed15
I20260926 03:49:33.359869 22922 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:49:33.376420 22922 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:49:33.378401 22922 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:49:33.392691 22922 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d1e086c926ed4853878bb394d16e533d: Generated new TSK 0
I20260926 03:49:33.393822 22922 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 03:49:33.402240 779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu pid 22839
I20260926 03:49:33.403570 22901 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:49:33.496763 779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu with pid 22839
2026-09-26T03:49:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (2037 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790394221564982-779-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:49:33 dist-test-slave-n2th krb5kdc[22938](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:49:33 dist-test-slave-n2th krb5kdc[22938](info): set up 2 sockets
Sep 26 03:49:33 dist-test-slave-n2th krb5kdc[22938](info): commencing operation
krb5kdc: starting...
W20260926 03:49:34.607300 779 mini_kdc.cc:121] Time spent starting KDC: real 1.074s user 0.003s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790394221564982-779-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790394221564982-779-0/krb5kdc/test-user.keytab.
Sep 26 03:49:34 dist-test-slave-n2th krb5kdc[22938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394574, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:49:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:49:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.194.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.194.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.194.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790394221564982-779-0/krb5kdc/kudu_127.0.194.254.keytab.
Entry for principal kudu/127.0.194.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790394221564982-779-0/krb5kdc/kudu_127.0.194.254.keytab.
I20260926 03:49:34.875860 779 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
/tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790394221564982-779-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790394221564982-779-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790394221564982-779-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790394221564982-779-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.194.254:42111
--webserver_interface=127.0.194.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790394221564982-779-0/krb5kdc/kudu_127.0.194.254.keytab
--principal=kudu/127.0.194.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.194.212:36959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.194.254:42111
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790394221564982-779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790394221564982-779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790394221564982-779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:49:35.219862 22951 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:49:35.220183 22951 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:49:35.220279 22951 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:49:35.227635 22951 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:49:35.227771 22951 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:49:35.227819 22951 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:49:35.227881 22951 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:49:35.227922 22951 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:49:35.237974 22951 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.194.212:36959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790394221564982-779-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790394221564982-779-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.194.254:42111
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790394221564982-779-0/krb5kdc/kudu_127.0.194.254.keytab
--principal=kudu/127.0.194.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.194.254:42111
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790394221564982-779-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.194.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790394221564982-779-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:49:35.239356 22951 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:49:35.240630 22951 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:49:35.253341 22956 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:49:35.253314 22959 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:49:35.253412 22957 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:49:35.260629 22951 server_base.cc:1034] running on GCE node
I20260926 03:49:35.261611 22951 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:49:35.263278 22951 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:49:35.264539 22951 hybrid_clock.cc:648] HybridClock initialized: now 1790394575264491 us; error 44 us; skew 500 ppm
I20260926 03:49:35.264909 22951 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:49:35 dist-test-slave-n2th krb5kdc[22938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394575, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:49:35.268335 22951 init.cc:377] Logged in from keytab as kudu/127.0.194.254@KRBTEST.COM (short username kudu)
I20260926 03:49:35.270004 22951 webserver.cc:460] Webserver started at http://127.0.194.254:38585/ using document root <none> and password file <none>
I20260926 03:49:35.270843 22951 fs_manager.cc:359] Metadata directory not provided
I20260926 03:49:35.270999 22951 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:49:35.271322 22951 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:49:35.274398 22951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790394221564982-779-0/minicluster-data/master-0/data/instance:
uuid: "5e07543f4ea344579eed85bf92e80b09"
format_stamp: "Formatted at 2026-09-26 03:49:35 on dist-test-slave-n2th"
server_key: "7647ca3eecef391da87c2afe0f86533f"
server_key_iv: "170029504da87231dab0287000d49db2"
server_key_version: "encryptionkey@0"
I20260926 03:49:35.275223 22951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790394221564982-779-0/minicluster-data/master-0/wal/instance:
uuid: "5e07543f4ea344579eed85bf92e80b09"
format_stamp: "Formatted at 2026-09-26 03:49:35 on dist-test-slave-n2th"
server_key: "7647ca3eecef391da87c2afe0f86533f"
server_key_iv: "170029504da87231dab0287000d49db2"
server_key_version: "encryptionkey@0"
I20260926 03:49:35.280623 22951 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 03:49:35.284343 22966 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:49:35.285784 22951 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 03:49:35.286026 22951 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790394221564982-779-0/minicluster-data/master-0/data,/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790394221564982-779-0/minicluster-data/master-0/wal
uuid: "5e07543f4ea344579eed85bf92e80b09"
format_stamp: "Formatted at 2026-09-26 03:49:35 on dist-test-slave-n2th"
server_key: "7647ca3eecef391da87c2afe0f86533f"
server_key_iv: "170029504da87231dab0287000d49db2"
server_key_version: "encryptionkey@0"
I20260926 03:49:35.286262 22951 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790394221564982-779-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790394221564982-779-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790394221564982-779-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:49:35.318071 22951 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:49:35.322711 22951 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:49:35.323061 22951 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:49:35.341785 22951 rpc_server.cc:307] RPC server started. Bound to: 127.0.194.254:42111
I20260926 03:49:35.341805 23017 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.194.254:42111 every 8 connection(s)
I20260926 03:49:35.343343 22951 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790394221564982-779-0/minicluster-data/master-0/data/info.pb
I20260926 03:49:35.347338 23018 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:49:35.352761 779 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu as pid 22951
I20260926 03:49:35.352934 779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790394221564982-779-0/minicluster-data/master-0/wal/instance
I20260926 03:49:35.353444 779 external_mini_cluster.cc:1428] Setting key 5c6de014c6c51337825600d425ac7915
I20260926 03:49:35.358702 23018 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e07543f4ea344579eed85bf92e80b09: Bootstrap starting.
I20260926 03:49:35.361974 23018 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5e07543f4ea344579eed85bf92e80b09: Neither blocks nor log segments found. Creating new log.
I20260926 03:49:35.363138 23018 log.cc:824] T 00000000000000000000000000000000 P 5e07543f4ea344579eed85bf92e80b09: Log is configured to *not* fsync() on all Append() calls
I20260926 03:49:35.366356 23018 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e07543f4ea344579eed85bf92e80b09: No bootstrap required, opened a new log
I20260926 03:49:35.367079 23021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:49:35.355473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38591 (local address 127.0.194.254:42111)
0926 03:49:35.356387 (+ 914us) server_negotiation.cc:206] Beginning negotiation
0926 03:49:35.356398 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 03:49:35.356448 (+ 50us) server_negotiation.cc:407] Connection header received
0926 03:49:35.358156 (+ 1708us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:49:35.358180 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:49:35.358592 (+ 412us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:49:35.359087 (+ 495us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:49:35.359636 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:49:35.360933 (+ 1297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:49:35.362327 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:49:35.363113 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:49:35.363315 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:49:35.364634 (+ 1319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:49:35.364641 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:49:35.364657 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:49:35.365202 (+ 545us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:49:35.365277 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:49:35.365302 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:49:35.365401 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0926 03:49:35.365660 (+ 259us) server_negotiation.cc:299] Negotiation successful
0926 03:49:35.365939 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":704,"thread_start_us":440,"threads_started":1}
I20260926 03:49:35.371234 779 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 03:49:35.375238 23018 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5e07543f4ea344579eed85bf92e80b09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e07543f4ea344579eed85bf92e80b09" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 42111 } }
I20260926 03:49:35.375766 23018 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5e07543f4ea344579eed85bf92e80b09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:49:35.375902 23018 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5e07543f4ea344579eed85bf92e80b09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e07543f4ea344579eed85bf92e80b09, State: Initialized, Role: FOLLOWER
I20260926 03:49:35.376657 23018 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5e07543f4ea344579eed85bf92e80b09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e07543f4ea344579eed85bf92e80b09" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 42111 } }
I20260926 03:49:35.376956 23018 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5e07543f4ea344579eed85bf92e80b09 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:49:35.377120 23018 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5e07543f4ea344579eed85bf92e80b09 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:49:35.377314 23018 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5e07543f4ea344579eed85bf92e80b09 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:49:35.380167 23018 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5e07543f4ea344579eed85bf92e80b09 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e07543f4ea344579eed85bf92e80b09" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 42111 } }
I20260926 03:49:35.380810 23018 leader_election.cc:302] T 00000000000000000000000000000000 P 5e07543f4ea344579eed85bf92e80b09 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5e07543f4ea344579eed85bf92e80b09; no voters:
I20260926 03:49:35.381531 23018 leader_election.cc:288] T 00000000000000000000000000000000 P 5e07543f4ea344579eed85bf92e80b09 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:49:35.381700 23028 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5e07543f4ea344579eed85bf92e80b09 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:49:35.383646 23021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:49:35.375753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39805 (local address 127.0.194.254:42111)
0926 03:49:35.375967 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0926 03:49:35.375974 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:49:35.376348 (+ 374us) server_negotiation.cc:407] Connection header received
0926 03:49:35.376537 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:49:35.376542 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:49:35.376622 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:49:35.376747 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:49:35.377351 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:49:35.378495 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:49:35.379950 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:49:35.380615 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:49:35.380744 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:49:35.381947 (+ 1203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:49:35.381955 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:49:35.381966 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:49:35.381994 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:49:35.382057 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:49:35.382082 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:49:35.382948 (+ 866us) server_negotiation.cc:1035] Waiting for connection context
0926 03:49:35.383013 (+ 65us) server_negotiation.cc:299] Negotiation successful
0926 03:49:35.383153 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 03:49:35.383723 23018 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5e07543f4ea344579eed85bf92e80b09 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:49:35.384960 23028 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5e07543f4ea344579eed85bf92e80b09 [term 1 LEADER]: Becoming Leader. State: Replica: 5e07543f4ea344579eed85bf92e80b09, State: Running, Role: LEADER
I20260926 03:49:35.385634 23028 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5e07543f4ea344579eed85bf92e80b09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e07543f4ea344579eed85bf92e80b09" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 42111 } }
I20260926 03:49:35.390491 23030 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e07543f4ea344579eed85bf92e80b09 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5e07543f4ea344579eed85bf92e80b09. Latest consensus state: current_term: 1 leader_uuid: "5e07543f4ea344579eed85bf92e80b09" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e07543f4ea344579eed85bf92e80b09" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 42111 } } }
I20260926 03:49:35.390498 23029 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e07543f4ea344579eed85bf92e80b09 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5e07543f4ea344579eed85bf92e80b09" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e07543f4ea344579eed85bf92e80b09" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 42111 } } }
I20260926 03:49:35.391036 23030 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e07543f4ea344579eed85bf92e80b09 [sys.catalog]: This master's current role is: LEADER
I20260926 03:49:35.391083 23029 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e07543f4ea344579eed85bf92e80b09 [sys.catalog]: This master's current role is: LEADER
W20260926 03:49:35.398124 23043 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5e07543f4ea344579eed85bf92e80b09: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 03:49:35.398300 23043 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 03:49:35.398420 23041 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:49:35.400183 23041 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:49:35.409256 23041 catalog_manager.cc:1347] Generated new cluster ID: d74b6b6dd9454963ac7f6bbd3c9d8ee5
I20260926 03:49:35.409394 23041 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:49:35.432932 23041 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:49:35.434309 23041 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:49:35.443100 23041 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5e07543f4ea344579eed85bf92e80b09: Generated new TSK 0
I20260926 03:49:35.444262 23041 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 03:49:35.455687 779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu pid 22951
W20260926 03:49:35.456804 23013 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:38591
W20260926 03:49:35.457515 779 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu pid 22951: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 03:49:35.457664 779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu with pid 22951
2026-09-26T03:49:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1960 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790394221564982-779-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:49:35 dist-test-slave-n2th krb5kdc[23049](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:49:35 dist-test-slave-n2th krb5kdc[23049](info): set up 2 sockets
Sep 26 03:49:35 dist-test-slave-n2th krb5kdc[23049](info): commencing operation
krb5kdc: starting...
W20260926 03:49:36.559315 779 mini_kdc.cc:121] Time spent starting KDC: real 1.065s user 0.006s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790394221564982-779-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790394221564982-779-0/krb5kdc/test-user.keytab.
Sep 26 03:49:36 dist-test-slave-n2th krb5kdc[23049](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394576, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:49:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:49:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.194.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.194.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.194.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790394221564982-779-0/krb5kdc/kudu_127.0.194.254.keytab.
Entry for principal kudu/127.0.194.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790394221564982-779-0/krb5kdc/kudu_127.0.194.254.keytab.
I20260926 03:49:36.828295 779 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
/tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790394221564982-779-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790394221564982-779-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790394221564982-779-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790394221564982-779-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.194.254:43889
--webserver_interface=127.0.194.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790394221564982-779-0/krb5kdc/kudu_127.0.194.254.keytab
--principal=kudu/127.0.194.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.194.212:39727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.194.254:43889
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790394221564982-779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790394221564982-779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790394221564982-779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:49:37.208038 23062 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:49:37.208421 23062 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:49:37.208597 23062 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:49:37.216099 23062 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:49:37.216286 23062 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:49:37.216403 23062 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:49:37.216506 23062 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:49:37.216609 23062 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:49:37.227885 23062 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.194.212:39727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790394221564982-779-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790394221564982-779-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.194.254:43889
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790394221564982-779-0/krb5kdc/kudu_127.0.194.254.keytab
--principal=kudu/127.0.194.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.194.254:43889
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790394221564982-779-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.194.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790394221564982-779-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:49:37.229246 23062 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:49:37.230628 23062 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:49:37.243599 23068 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:49:37.243671 23070 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:49:37.243671 23067 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:49:37.244886 23062 server_base.cc:1034] running on GCE node
I20260926 03:49:37.245746 23062 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:49:37.247565 23062 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:49:37.248795 23062 hybrid_clock.cc:648] HybridClock initialized: now 1790394577248727 us; error 67 us; skew 500 ppm
I20260926 03:49:37.249230 23062 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:49:37 dist-test-slave-n2th krb5kdc[23049](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394577, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:49:37.252735 23062 init.cc:377] Logged in from keytab as kudu/127.0.194.254@KRBTEST.COM (short username kudu)
I20260926 03:49:37.254298 23062 webserver.cc:460] Webserver started at http://127.0.194.254:45391/ using document root <none> and password file <none>
I20260926 03:49:37.255101 23062 fs_manager.cc:359] Metadata directory not provided
I20260926 03:49:37.255223 23062 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:49:37.255483 23062 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:49:37.258670 23062 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790394221564982-779-0/minicluster-data/master-0/data/instance:
uuid: "a32739f95d3147e194c220400174e5bc"
format_stamp: "Formatted at 2026-09-26 03:49:37 on dist-test-slave-n2th"
server_key: "0f54c430df6c586ca910f8e41ceb9a7b"
server_key_iv: "a7a2260795aa5942a44c452f8733f1d8"
server_key_version: "encryptionkey@0"
I20260926 03:49:37.259675 23062 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790394221564982-779-0/minicluster-data/master-0/wal/instance:
uuid: "a32739f95d3147e194c220400174e5bc"
format_stamp: "Formatted at 2026-09-26 03:49:37 on dist-test-slave-n2th"
server_key: "0f54c430df6c586ca910f8e41ceb9a7b"
server_key_iv: "a7a2260795aa5942a44c452f8733f1d8"
server_key_version: "encryptionkey@0"
I20260926 03:49:37.265479 23062 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260926 03:49:37.269541 23077 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:49:37.271148 23062 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260926 03:49:37.271366 23062 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790394221564982-779-0/minicluster-data/master-0/data,/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790394221564982-779-0/minicluster-data/master-0/wal
uuid: "a32739f95d3147e194c220400174e5bc"
format_stamp: "Formatted at 2026-09-26 03:49:37 on dist-test-slave-n2th"
server_key: "0f54c430df6c586ca910f8e41ceb9a7b"
server_key_iv: "a7a2260795aa5942a44c452f8733f1d8"
server_key_version: "encryptionkey@0"
I20260926 03:49:37.271592 23062 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790394221564982-779-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790394221564982-779-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790394221564982-779-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:49:37.278076 23062 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:49:37.282963 23062 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:49:37.283304 23062 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:49:37.303299 23062 rpc_server.cc:307] RPC server started. Bound to: 127.0.194.254:43889
I20260926 03:49:37.303335 23128 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.194.254:43889 every 8 connection(s)
I20260926 03:49:37.304811 23062 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790394221564982-779-0/minicluster-data/master-0/data/info.pb
I20260926 03:49:37.308700 779 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu as pid 23062
I20260926 03:49:37.308889 779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790394221564982-779-0/minicluster-data/master-0/wal/instance
I20260926 03:49:37.309062 23129 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:49:37.309448 779 external_mini_cluster.cc:1428] Setting key 257eee1af5467246833ad2ce36c1b051
I20260926 03:49:37.318403 23132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:49:37.311641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54333 (local address 127.0.194.254:43889)
0926 03:49:37.312633 (+ 992us) server_negotiation.cc:206] Beginning negotiation
0926 03:49:37.312649 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 03:49:37.312699 (+ 50us) server_negotiation.cc:407] Connection header received
0926 03:49:37.314493 (+ 1794us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:49:37.314520 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:49:37.315052 (+ 532us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:49:37.315459 (+ 407us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:49:37.316669 (+ 1210us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:49:37.316676 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:49:37.316691 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:49:37.316719 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:49:37.316788 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:49:37.316812 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:49:37.317041 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0926 03:49:37.317157 (+ 116us) server_negotiation.cc:299] Negotiation successful
0926 03:49:37.317439 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":735,"thread_start_us":489,"threads_started":1}
I20260926 03:49:37.322583 779 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 03:49:37.323791 23129 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a32739f95d3147e194c220400174e5bc: Bootstrap starting.
I20260926 03:49:37.327477 23129 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a32739f95d3147e194c220400174e5bc: Neither blocks nor log segments found. Creating new log.
I20260926 03:49:37.328776 23129 log.cc:824] T 00000000000000000000000000000000 P a32739f95d3147e194c220400174e5bc: Log is configured to *not* fsync() on all Append() calls
I20260926 03:49:37.329879 23132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:49:37.327094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49155 (local address 127.0.194.254:43889)
0926 03:49:37.327309 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0926 03:49:37.327317 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:49:37.327771 (+ 454us) server_negotiation.cc:407] Connection header received
0926 03:49:37.327941 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:49:37.327949 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:49:37.328031 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:49:37.328145 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:49:37.329129 (+ 984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:49:37.329135 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:49:37.329139 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:49:37.329160 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:49:37.329206 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:49:37.329231 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:49:37.329380 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0926 03:49:37.329479 (+ 99us) server_negotiation.cc:299] Negotiation successful
0926 03:49:37.329557 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 03:49:37.332049 23129 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a32739f95d3147e194c220400174e5bc: No bootstrap required, opened a new log
I20260926 03:49:37.340004 23129 raft_consensus.cc:348] T 00000000000000000000000000000000 P a32739f95d3147e194c220400174e5bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a32739f95d3147e194c220400174e5bc" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 43889 } }
I20260926 03:49:37.340426 23129 raft_consensus.cc:374] T 00000000000000000000000000000000 P a32739f95d3147e194c220400174e5bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:49:37.340559 23129 raft_consensus.cc:729] T 00000000000000000000000000000000 P a32739f95d3147e194c220400174e5bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a32739f95d3147e194c220400174e5bc, State: Initialized, Role: FOLLOWER
I20260926 03:49:37.341224 23129 consensus_queue.cc:260] T 00000000000000000000000000000000 P a32739f95d3147e194c220400174e5bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a32739f95d3147e194c220400174e5bc" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 43889 } }
I20260926 03:49:37.341457 23129 raft_consensus.cc:388] T 00000000000000000000000000000000 P a32739f95d3147e194c220400174e5bc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:49:37.341600 23129 raft_consensus.cc:482] T 00000000000000000000000000000000 P a32739f95d3147e194c220400174e5bc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:49:37.341774 23129 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a32739f95d3147e194c220400174e5bc [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:49:37.344354 23129 raft_consensus.cc:504] T 00000000000000000000000000000000 P a32739f95d3147e194c220400174e5bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a32739f95d3147e194c220400174e5bc" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 43889 } }
I20260926 03:49:37.344887 23129 leader_election.cc:302] T 00000000000000000000000000000000 P a32739f95d3147e194c220400174e5bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a32739f95d3147e194c220400174e5bc; no voters:
I20260926 03:49:37.345499 23129 leader_election.cc:288] T 00000000000000000000000000000000 P a32739f95d3147e194c220400174e5bc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:49:37.345656 23139 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a32739f95d3147e194c220400174e5bc [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:49:37.346341 23139 raft_consensus.cc:686] T 00000000000000000000000000000000 P a32739f95d3147e194c220400174e5bc [term 1 LEADER]: Becoming Leader. State: Replica: a32739f95d3147e194c220400174e5bc, State: Running, Role: LEADER
I20260926 03:49:37.347014 23139 consensus_queue.cc:237] T 00000000000000000000000000000000 P a32739f95d3147e194c220400174e5bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a32739f95d3147e194c220400174e5bc" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 43889 } }
I20260926 03:49:37.347715 23129 sys_catalog.cc:564] T 00000000000000000000000000000000 P a32739f95d3147e194c220400174e5bc [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:49:37.349799 23141 sys_catalog.cc:455] T 00000000000000000000000000000000 P a32739f95d3147e194c220400174e5bc [sys.catalog]: SysCatalogTable state changed. Reason: New leader a32739f95d3147e194c220400174e5bc. Latest consensus state: current_term: 1 leader_uuid: "a32739f95d3147e194c220400174e5bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a32739f95d3147e194c220400174e5bc" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 43889 } } }
I20260926 03:49:37.350195 23141 sys_catalog.cc:458] T 00000000000000000000000000000000 P a32739f95d3147e194c220400174e5bc [sys.catalog]: This master's current role is: LEADER
I20260926 03:49:37.351151 23144 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:49:37.351073 23140 sys_catalog.cc:455] T 00000000000000000000000000000000 P a32739f95d3147e194c220400174e5bc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a32739f95d3147e194c220400174e5bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a32739f95d3147e194c220400174e5bc" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 43889 } } }
I20260926 03:49:37.351397 23140 sys_catalog.cc:458] T 00000000000000000000000000000000 P a32739f95d3147e194c220400174e5bc [sys.catalog]: This master's current role is: LEADER
I20260926 03:49:37.357558 23144 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:49:37.366947 23144 catalog_manager.cc:1347] Generated new cluster ID: 93d2bb5403e8479491c2ea12820261e8
I20260926 03:49:37.367074 23144 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:49:37.387596 23144 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:49:37.388077 23144 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:49:37.405462 23144 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a32739f95d3147e194c220400174e5bc: Generated new TSK 0
I20260926 03:49:37.406451 23144 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 03:49:37.418354 779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu pid 23062
W20260926 03:49:37.419873 23124 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:54333
W20260926 03:49:37.420656 779 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu pid 23062: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 03:49:37.420830 779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu with pid 23062
2026-09-26T03:49:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1963 ms)
[----------] 18 tests from AuthTokenIssuingTest (35806 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790394221564982-779-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:49:37 dist-test-slave-n2th krb5kdc[23160](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:49:37 dist-test-slave-n2th krb5kdc[23160](info): set up 2 sockets
Sep 26 03:49:37 dist-test-slave-n2th krb5kdc[23160](info): commencing operation
krb5kdc: starting...
W20260926 03:49:38.527527 779 mini_kdc.cc:121] Time spent starting KDC: real 1.070s user 0.004s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790394221564982-779-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790394221564982-779-0/krb5kdc/test-user.keytab.
Sep 26 03:49:38 dist-test-slave-n2th krb5kdc[23160](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394578, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:49:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:49:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.194.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.194.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.194.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790394221564982-779-0/krb5kdc/kudu_127.0.194.254.keytab.
Entry for principal kudu/127.0.194.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790394221564982-779-0/krb5kdc/kudu_127.0.194.254.keytab.
I20260926 03:49:38.799870 779 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
/tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790394221564982-779-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790394221564982-779-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790394221564982-779-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790394221564982-779-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.194.254:45639
--webserver_interface=127.0.194.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790394221564982-779-0/krb5kdc/kudu_127.0.194.254.keytab
--principal=kudu/127.0.194.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.194.212:44963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.194.254:45639,127.0.194.253:34749,127.0.194.252:43305
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790394221564982-779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790394221564982-779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790394221564982-779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:49:39.140051 23173 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 03:49:39.140393 23173 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:49:39.140458 23173 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:49:39.140591 23173 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:49:39.148170 23173 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:49:39.148275 23173 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:49:39.148363 23173 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:49:39.148466 23173 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:49:39.148577 23173 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:49:39.159027 23173 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.194.212:44963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790394221564982-779-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790394221564982-779-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.194.254:45639,127.0.194.253:34749,127.0.194.252:43305
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790394221564982-779-0/krb5kdc/kudu_127.0.194.254.keytab
--principal=kudu/127.0.194.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.194.254:45639
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790394221564982-779-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.194.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790394221564982-779-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:49:39.160385 23173 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:49:39.161732 23173 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:49:39.173902 23178 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:49:39.173970 23181 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:49:39.173935 23179 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:49:39.174700 23173 server_base.cc:1034] running on GCE node
I20260926 03:49:39.175566 23173 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:49:39.176959 23173 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:49:39.178185 23173 hybrid_clock.cc:648] HybridClock initialized: now 1790394579178117 us; error 42 us; skew 500 ppm
I20260926 03:49:39.178534 23173 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:49:39 dist-test-slave-n2th krb5kdc[23160](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394579, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:49:39.186362 23173 init.cc:377] Logged in from keytab as kudu/127.0.194.254@KRBTEST.COM (short username kudu)
I20260926 03:49:39.188074 23173 webserver.cc:460] Webserver started at http://127.0.194.254:34815/ using document root <none> and password file <none>
I20260926 03:49:39.188809 23173 fs_manager.cc:359] Metadata directory not provided
I20260926 03:49:39.188948 23173 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:49:39.189286 23173 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:49:39.192456 23173 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790394221564982-779-0/minicluster-data/master-0/data/instance:
uuid: "9139e44175f74472bddc1f3e3c159754"
format_stamp: "Formatted at 2026-09-26 03:49:39 on dist-test-slave-n2th"
server_key: "c1ace97330a3e949f68534924bc6f707"
server_key_iv: "e3d7235522145785d10e6848b91c0420"
server_key_version: "encryptionkey@0"
I20260926 03:49:39.193266 23173 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790394221564982-779-0/minicluster-data/master-0/wal/instance:
uuid: "9139e44175f74472bddc1f3e3c159754"
format_stamp: "Formatted at 2026-09-26 03:49:39 on dist-test-slave-n2th"
server_key: "c1ace97330a3e949f68534924bc6f707"
server_key_iv: "e3d7235522145785d10e6848b91c0420"
server_key_version: "encryptionkey@0"
I20260926 03:49:39.198716 23173 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260926 03:49:39.202425 23188 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:49:39.203891 23173 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260926 03:49:39.204106 23173 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790394221564982-779-0/minicluster-data/master-0/data,/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790394221564982-779-0/minicluster-data/master-0/wal
uuid: "9139e44175f74472bddc1f3e3c159754"
format_stamp: "Formatted at 2026-09-26 03:49:39 on dist-test-slave-n2th"
server_key: "c1ace97330a3e949f68534924bc6f707"
server_key_iv: "e3d7235522145785d10e6848b91c0420"
server_key_version: "encryptionkey@0"
I20260926 03:49:39.204337 23173 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790394221564982-779-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790394221564982-779-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790394221564982-779-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:49:39.235312 23173 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:49:39.239589 23173 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:49:39.239884 23173 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:49:39.259001 23173 rpc_server.cc:307] RPC server started. Bound to: 127.0.194.254:45639
I20260926 03:49:39.259020 23239 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.194.254:45639 every 8 connection(s)
I20260926 03:49:39.260437 23173 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790394221564982-779-0/minicluster-data/master-0/data/info.pb
I20260926 03:49:39.264257 23240 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:49:39.266557 779 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu as pid 23173
I20260926 03:49:39.266817 779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790394221564982-779-0/minicluster-data/master-0/wal/instance
I20260926 03:49:39.267480 779 external_mini_cluster.cc:1428] Setting key eb86c3591a89c363dcaf1eb861ecdd2d
I20260926 03:49:39.271191 23240 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.194.254" port: 45639 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 03:49:39 dist-test-slave-n2th krb5kdc[23160](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394579, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.254@KRBTEST.COM for kudu/127.0.194.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.194.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.194.253@KRBTEST.COM" created.
I20260926 03:49:39.297785 23243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:49:39.274903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37542 (local address 127.0.194.254:45639)
0926 03:49:39.276855 (+ 1952us) server_negotiation.cc:206] Beginning negotiation
0926 03:49:39.276865 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 03:49:39.277048 (+ 183us) server_negotiation.cc:407] Connection header received
0926 03:49:39.279197 (+ 2149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:49:39.279207 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:49:39.279538 (+ 331us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:49:39.279759 (+ 221us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:49:39.280920 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:49:39.282854 (+ 1934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:49:39.284286 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:49:39.285299 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:49:39.285495 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:49:39.289028 (+ 3533us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:49:39.289043 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:49:39.289066 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:49:39.289116 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:49:39.293247 (+ 4131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:49:39.294101 (+ 854us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:49:39.294108 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:49:39.294110 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:49:39.294264 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:49:39.294705 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:49:39.294708 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:49:39.294710 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:49:39.295305 (+ 595us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:49:39.295493 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0926 03:49:39.296113 (+ 620us) server_negotiation.cc:299] Negotiation successful
0926 03:49:39.296549 (+ 436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":1755,"spinlock_wait_cycles":37632,"thread_start_us":1522,"threads_started":1}
I20260926 03:49:39.297823 23242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:49:39.274903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:45639 (local address 127.0.0.1:37542)
0926 03:49:39.276812 (+ 1909us) negotiation.cc:107] Waiting for socket to connect
0926 03:49:39.276875 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0926 03:49:39.278713 (+ 1838us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:49:39.279785 (+ 1072us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:49:39.279790 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:49:39.280367 (+ 577us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:49:39.280705 (+ 338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:49:39.280721 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:49:39.283009 (+ 2288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:49:39.283013 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:49:39.284139 (+ 1126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:49:39.284151 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:49:39.285519 (+ 1368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:49:39.285524 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:49:39.285648 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:49:39.286372 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:49:39.286396 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:49:39.288740 (+ 2344us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:49:39.293421 (+ 4681us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:49:39.293433 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:49:39.293447 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:49:39.293955 (+ 508us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:49:39.294369 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:49:39.294373 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:49:39.294375 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:49:39.294565 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:49:39.295459 (+ 894us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:49:39.295466 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:49:39.295755 (+ 289us) client_negotiation.cc:769] Sending connection context
0926 03:49:39.295986 (+ 231us) client_negotiation.cc:240] Negotiation successful
0926 03:49:39.296724 (+ 738us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1573,"mutex_wait_us":208,"thread_start_us":386,"threads_started":1}
I20260926 03:49:39.300897 23240 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.194.253" port: 34749 } has no permanent_uuid. Determining permanent_uuid...
I20260926 03:49:39.302111 23242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:49:39.301590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.253:34749 (local address 127.0.0.1:52396)
0926 03:49:39.301801 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0926 03:49:39.301943 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.253:34749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260926 03:49:39.302390 23190 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.194.253:34749: connect: Connection refused (error 111)
I20260926 03:49:39.304148 23242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:49:39.303648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.253:34749 (local address 127.0.0.1:52398)
0926 03:49:39.303854 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0926 03:49:39.303953 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.253:34749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260926 03:49:39.304347 23240 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.194.253:34749: Network error: Client connection negotiation failed: client connection to 127.0.194.253:34749: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.194.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790394221564982-779-0/krb5kdc/kudu_127.0.194.253.keytab.
Entry for principal kudu/127.0.194.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790394221564982-779-0/krb5kdc/kudu_127.0.194.253.keytab.
I20260926 03:49:39.329905 779 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
/tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790394221564982-779-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790394221564982-779-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790394221564982-779-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790394221564982-779-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.194.253:34749
--webserver_interface=127.0.194.253
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790394221564982-779-0/krb5kdc/kudu_127.0.194.253.keytab
--principal=kudu/127.0.194.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.194.212:44963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.194.254:45639,127.0.194.253:34749,127.0.194.252:43305
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790394221564982-779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790394221564982-779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790394221564982-779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 03:49:39.353708 23240 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.194.253" port: 34749 } attempt: 1
I20260926 03:49:39.355026 23242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:49:39.354507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.253:34749 (local address 127.0.0.1:52400)
0926 03:49:39.354720 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0926 03:49:39.354897 (+ 177us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.253:34749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260926 03:49:39.356005 23242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:49:39.355656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.253:34749 (local address 127.0.0.1:52412)
0926 03:49:39.355815 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0926 03:49:39.355881 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.253:34749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260926 03:49:39.356191 23240 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.194.253:34749: Network error: Client connection negotiation failed: client connection to 127.0.194.253:34749: connect: Connection refused (error 111)
I20260926 03:49:39.424463 23240 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.194.253" port: 34749 } attempt: 2
I20260926 03:49:39.425815 23242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:49:39.425324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.253:34749 (local address 127.0.0.1:52424)
0926 03:49:39.425577 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0926 03:49:39.425670 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.253:34749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260926 03:49:39.426985 23242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:49:39.426514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.253:34749 (local address 127.0.0.1:52434)
0926 03:49:39.426663 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0926 03:49:39.426827 (+ 164us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.253:34749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260926 03:49:39.427161 23240 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.194.253:34749: Network error: Client connection negotiation failed: client connection to 127.0.194.253:34749: connect: Connection refused (error 111)
I20260926 03:49:39.518440 23240 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.194.253" port: 34749 } attempt: 3
I20260926 03:49:39.519641 23242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:49:39.519185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.253:34749 (local address 127.0.0.1:52450)
0926 03:49:39.519390 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0926 03:49:39.519506 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.253:34749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260926 03:49:39.520674 23242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:49:39.520301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.253:34749 (local address 127.0.0.1:52464)
0926 03:49:39.520472 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0926 03:49:39.520538 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.253:34749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260926 03:49:39.520867 23240 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.194.253:34749: Network error: Client connection negotiation failed: client connection to 127.0.194.253:34749: connect: Connection refused (error 111)
I20260926 03:49:39.664124 23240 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.194.253" port: 34749 } attempt: 4
I20260926 03:49:39.665326 23242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:49:39.664771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.253:34749 (local address 127.0.0.1:52466)
0926 03:49:39.665095 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0926 03:49:39.665187 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.253:34749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152}
I20260926 03:49:39.666322 23242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:49:39.665964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.253:34749 (local address 127.0.0.1:52474)
0926 03:49:39.666113 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0926 03:49:39.666202 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.253:34749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260926 03:49:39.666519 23240 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.194.253:34749: Network error: Client connection negotiation failed: client connection to 127.0.194.253:34749: connect: Connection refused (error 111)
W20260926 03:49:39.668301 23247 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 03:49:39.668666 23247 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:49:39.668790 23247 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:49:39.668923 23247 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:49:39.676477 23247 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:49:39.676609 23247 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:49:39.676740 23247 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:49:39.676829 23247 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:49:39.676941 23247 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:49:39.687258 23247 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.194.212:44963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790394221564982-779-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790394221564982-779-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.194.254:45639,127.0.194.253:34749,127.0.194.252:43305
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790394221564982-779-0/krb5kdc/kudu_127.0.194.253.keytab
--principal=kudu/127.0.194.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.194.253:34749
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790394221564982-779-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.194.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790394221564982-779-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:49:39.688637 23247 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:49:39.690021 23247 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:49:39.702193 23253 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:49:39.702582 23255 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:49:39.702805 23252 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:49:39.703519 23247 server_base.cc:1034] running on GCE node
I20260926 03:49:39.704373 23247 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:49:39.705873 23247 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:49:39.707132 23247 hybrid_clock.cc:648] HybridClock initialized: now 1790394579707061 us; error 46 us; skew 500 ppm
I20260926 03:49:39.707499 23247 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:49:39 dist-test-slave-n2th krb5kdc[23160](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394579, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:49:39.710778 23247 init.cc:377] Logged in from keytab as kudu/127.0.194.253@KRBTEST.COM (short username kudu)
I20260926 03:49:39.712473 23247 webserver.cc:460] Webserver started at http://127.0.194.253:46239/ using document root <none> and password file <none>
I20260926 03:49:39.713644 23247 fs_manager.cc:359] Metadata directory not provided
I20260926 03:49:39.713780 23247 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:49:39.714133 23247 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:49:39.718973 23247 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790394221564982-779-0/minicluster-data/master-1/data/instance:
uuid: "20b4675e77d1407bba747dca53ba150e"
format_stamp: "Formatted at 2026-09-26 03:49:39 on dist-test-slave-n2th"
server_key: "1ec371faca81046848baad63d7c2bd37"
server_key_iv: "4aaf213c05ad986c791514dd0cf8a16d"
server_key_version: "encryptionkey@0"
I20260926 03:49:39.720127 23247 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790394221564982-779-0/minicluster-data/master-1/wal/instance:
uuid: "20b4675e77d1407bba747dca53ba150e"
format_stamp: "Formatted at 2026-09-26 03:49:39 on dist-test-slave-n2th"
server_key: "1ec371faca81046848baad63d7c2bd37"
server_key_iv: "4aaf213c05ad986c791514dd0cf8a16d"
server_key_version: "encryptionkey@0"
I20260926 03:49:39.726089 23247 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260926 03:49:39.729800 23262 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:49:39.731303 23247 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 03:49:39.731498 23247 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790394221564982-779-0/minicluster-data/master-1/data,/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790394221564982-779-0/minicluster-data/master-1/wal
uuid: "20b4675e77d1407bba747dca53ba150e"
format_stamp: "Formatted at 2026-09-26 03:49:39 on dist-test-slave-n2th"
server_key: "1ec371faca81046848baad63d7c2bd37"
server_key_iv: "4aaf213c05ad986c791514dd0cf8a16d"
server_key_version: "encryptionkey@0"
I20260926 03:49:39.731743 23247 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790394221564982-779-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790394221564982-779-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790394221564982-779-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:49:39.753280 23247 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:49:39.757465 23247 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:49:39.757764 23247 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:49:39.776410 23247 rpc_server.cc:307] RPC server started. Bound to: 127.0.194.253:34749
I20260926 03:49:39.776430 23313 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.194.253:34749 every 8 connection(s)
I20260926 03:49:39.777770 23247 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790394221564982-779-0/minicluster-data/master-1/data/info.pb
I20260926 03:49:39.781545 23314 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:49:39.786085 779 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu as pid 23247
I20260926 03:49:39.786267 779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790394221564982-779-0/minicluster-data/master-1/wal/instance
I20260926 03:49:39.786868 779 external_mini_cluster.cc:1428] Setting key 34e95bd0e0ab2e4262908749fde8971d
I20260926 03:49:39.789420 23314 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.194.254" port: 45639 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 03:49:39 dist-test-slave-n2th krb5kdc[23160](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394579, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.253@KRBTEST.COM for kudu/127.0.194.254@KRBTEST.COM
I20260926 03:49:39.813426 23243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:49:39.792987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37558 (local address 127.0.194.254:45639)
0926 03:49:39.793241 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0926 03:49:39.793250 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:49:39.794490 (+ 1240us) server_negotiation.cc:407] Connection header received
0926 03:49:39.796429 (+ 1939us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:49:39.796435 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:49:39.796529 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:49:39.796648 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:49:39.798535 (+ 1887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:49:39.799347 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:49:39.801806 (+ 2459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:49:39.802232 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:49:39.802335 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:49:39.806658 (+ 4323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:49:39.806663 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:49:39.806666 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:49:39.806700 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:49:39.809751 (+ 3051us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:49:39.810882 (+ 1131us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:49:39.810891 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:49:39.810897 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:49:39.811038 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:49:39.811572 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:49:39.811575 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:49:39.811577 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:49:39.811870 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:49:39.811994 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0926 03:49:39.812982 (+ 988us) server_negotiation.cc:299] Negotiation successful
0926 03:49:39.813193 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
WARNING: no policy specified for kudu/127.0.194.252@KRBTEST.COM; defaulting to no policy
I20260926 03:49:39.814975 23316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:49:39.793492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:45639 (local address 127.0.0.1:37558)
0926 03:49:39.794304 (+ 812us) negotiation.cc:107] Waiting for socket to connect
0926 03:49:39.794357 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 03:49:39.796159 (+ 1802us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:49:39.796877 (+ 718us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:49:39.796885 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:49:39.797519 (+ 634us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:49:39.798289 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:49:39.798315 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:49:39.799520 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:49:39.799527 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:49:39.801549 (+ 2022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:49:39.801566 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:49:39.802487 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:49:39.802496 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:49:39.802844 (+ 348us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:49:39.803581 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:49:39.803620 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:49:39.806394 (+ 2774us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:49:39.809949 (+ 3555us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:49:39.809964 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:49:39.809987 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:49:39.810718 (+ 731us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:49:39.811161 (+ 443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:49:39.811169 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:49:39.811175 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:49:39.811450 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:49:39.812005 (+ 555us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:49:39.812014 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:49:39.812529 (+ 515us) client_negotiation.cc:769] Sending connection context
0926 03:49:39.812882 (+ 353us) client_negotiation.cc:240] Negotiation successful
0926 03:49:39.813476 (+ 594us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":562,"thread_start_us":352,"threads_started":1}
I20260926 03:49:39.816654 23314 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.194.253" port: 34749 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.194.252@KRBTEST.COM" created.
Sep 26 03:49:39 dist-test-slave-n2th krb5kdc[23160](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394579, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.253@KRBTEST.COM for kudu/127.0.194.253@KRBTEST.COM
I20260926 03:49:39.833477 23318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:49:39.817741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52484 (local address 127.0.194.253:34749)
0926 03:49:39.818437 (+ 696us) server_negotiation.cc:206] Beginning negotiation
0926 03:49:39.818444 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:49:39.818470 (+ 26us) server_negotiation.cc:407] Connection header received
0926 03:49:39.818514 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:49:39.818519 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:49:39.818706 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:49:39.818928 (+ 222us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:49:39.819894 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:49:39.820941 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:49:39.822711 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:49:39.823404 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:49:39.823498 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:49:39.827363 (+ 3865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:49:39.827383 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:49:39.827406 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:49:39.827453 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:49:39.830351 (+ 2898us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:49:39.831301 (+ 950us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:49:39.831308 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:49:39.831311 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:49:39.831425 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:49:39.831848 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:49:39.831851 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:49:39.831853 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:49:39.832200 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:49:39.832366 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0926 03:49:39.832806 (+ 440us) server_negotiation.cc:299] Negotiation successful
0926 03:49:39.833159 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":142,"server-negotiator.queue_time_us":544,"thread_start_us":406,"threads_started":1}
I20260926 03:49:39.833576 23316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:49:39.817586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.253:34749 (local address 127.0.0.1:52484)
0926 03:49:39.817827 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0926 03:49:39.817848 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 03:49:39.817963 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:49:39.818958 (+ 995us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:49:39.818965 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:49:39.819488 (+ 523us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:49:39.819722 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:49:39.819737 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:49:39.821082 (+ 1345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:49:39.821090 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:49:39.822565 (+ 1475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:49:39.822583 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:49:39.823583 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:49:39.823592 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:49:39.823806 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:49:39.824426 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:49:39.824454 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:49:39.827131 (+ 2677us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:49:39.830526 (+ 3395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:49:39.830540 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:49:39.830547 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:49:39.831174 (+ 627us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:49:39.831538 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:49:39.831546 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:49:39.831552 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:49:39.831736 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:49:39.832343 (+ 607us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:49:39.832351 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:49:39.832520 (+ 169us) client_negotiation.cc:769] Sending connection context
0926 03:49:39.832686 (+ 166us) client_negotiation.cc:240] Negotiation successful
0926 03:49:39.833036 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66}
I20260926 03:49:39.836256 23314 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.194.252" port: 43305 } has no permanent_uuid. Determining permanent_uuid...
I20260926 03:49:39.837723 23316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:49:39.837154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.252:43305 (local address 127.0.0.1:51170)
0926 03:49:39.837379 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0926 03:49:39.837536 (+ 157us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.252:43305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
W20260926 03:49:39.838085 23265 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.194.252:43305: connect: Connection refused (error 111)
I20260926 03:49:39.840044 23316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:49:39.839663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.252:43305 (local address 127.0.0.1:51178)
0926 03:49:39.839833 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0926 03:49:39.839921 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.252:43305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260926 03:49:39.840315 23314 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.194.252:43305: Network error: Client connection negotiation failed: client connection to 127.0.194.252:43305: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.194.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790394221564982-779-0/krb5kdc/kudu_127.0.194.252.keytab.
Entry for principal kudu/127.0.194.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790394221564982-779-0/krb5kdc/kudu_127.0.194.252.keytab.
I20260926 03:49:39.852046 779 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
/tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790394221564982-779-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790394221564982-779-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790394221564982-779-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790394221564982-779-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.194.252:43305
--webserver_interface=127.0.194.252
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790394221564982-779-0/krb5kdc/kudu_127.0.194.252.keytab
--principal=kudu/127.0.194.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.194.212:44963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.194.254:45639,127.0.194.253:34749,127.0.194.252:43305
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790394221564982-779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790394221564982-779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790394221564982-779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 03:49:39.889679 23314 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.194.252" port: 43305 } attempt: 1
I20260926 03:49:39.891010 23316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:49:39.890442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.252:43305 (local address 127.0.0.1:51186)
0926 03:49:39.890697 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0926 03:49:39.890857 (+ 160us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.252:43305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260926 03:49:39.892050 23316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:49:39.891705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.252:43305 (local address 127.0.0.1:51196)
0926 03:49:39.891877 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0926 03:49:39.891949 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.252:43305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260926 03:49:39.892239 23314 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.194.252:43305: Network error: Client connection negotiation failed: client connection to 127.0.194.252:43305: connect: Connection refused (error 111)
I20260926 03:49:39.960446 23314 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.194.252" port: 43305 } attempt: 2
I20260926 03:49:39.961704 23316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:49:39.961128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.252:43305 (local address 127.0.0.1:51212)
0926 03:49:39.961379 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0926 03:49:39.961496 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.252:43305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260926 03:49:39.962848 23316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:49:39.962368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.252:43305 (local address 127.0.0.1:51220)
0926 03:49:39.962568 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0926 03:49:39.962673 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.252:43305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260926 03:49:39.963114 23314 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.194.252:43305: Network error: Client connection negotiation failed: client connection to 127.0.194.252:43305: connect: Connection refused (error 111)
I20260926 03:49:39.965772 23240 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.194.253" port: 34749 } attempt: 5
Sep 26 03:49:39 dist-test-slave-n2th krb5kdc[23160](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394579, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.254@KRBTEST.COM for kudu/127.0.194.253@KRBTEST.COM
I20260926 03:49:39.980206 23318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:49:39.966654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52488 (local address 127.0.194.253:34749)
0926 03:49:39.966829 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0926 03:49:39.966837 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:49:39.966860 (+ 23us) server_negotiation.cc:407] Connection header received
0926 03:49:39.967130 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:49:39.967135 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:49:39.967218 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:49:39.967330 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:49:39.968067 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:49:39.969150 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:49:39.970339 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:49:39.971023 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:49:39.971155 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:49:39.974417 (+ 3262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:49:39.974422 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:49:39.974426 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:49:39.974458 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:49:39.977115 (+ 2657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:49:39.977973 (+ 858us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:49:39.977980 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:49:39.977986 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:49:39.978147 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:49:39.978663 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:49:39.978670 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:49:39.978674 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:49:39.979227 (+ 553us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:49:39.979342 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0926 03:49:39.979806 (+ 464us) server_negotiation.cc:299] Negotiation successful
0926 03:49:39.979977 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 03:49:39.980353 23242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:49:39.966523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.253:34749 (local address 127.0.0.1:52488)
0926 03:49:39.966768 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0926 03:49:39.966790 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 03:49:39.966945 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:49:39.967375 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:49:39.967379 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:49:39.967739 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:49:39.967912 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:49:39.967920 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:49:39.969317 (+ 1397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:49:39.969322 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:49:39.970177 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:49:39.970189 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:49:39.971181 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:49:39.971186 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:49:39.971354 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:49:39.972007 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:49:39.972024 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:49:39.974225 (+ 2201us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:49:39.977267 (+ 3042us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:49:39.977273 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:49:39.977277 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:49:39.977817 (+ 540us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:49:39.978282 (+ 465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:49:39.978287 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:49:39.978290 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:49:39.978516 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:49:39.979350 (+ 834us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:49:39.979355 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:49:39.979538 (+ 183us) client_negotiation.cc:769] Sending connection context
0926 03:49:39.979718 (+ 180us) client_negotiation.cc:240] Negotiation successful
0926 03:49:39.980015 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":78}
I20260926 03:49:39.981587 23240 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.194.252" port: 43305 } has no permanent_uuid. Determining permanent_uuid...
I20260926 03:49:39.982720 23242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:49:39.982326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.252:43305 (local address 127.0.0.1:51224)
0926 03:49:39.982504 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0926 03:49:39.982592 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.252:43305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260926 03:49:39.983702 23242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:49:39.983390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.252:43305 (local address 127.0.0.1:51228)
0926 03:49:39.983539 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0926 03:49:39.983617 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.252:43305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260926 03:49:39.983891 23240 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.194.252:43305: Network error: Client connection negotiation failed: client connection to 127.0.194.252:43305: connect: Connection refused (error 111)
I20260926 03:49:40.035156 23240 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.194.252" port: 43305 } attempt: 1
I20260926 03:49:40.036403 23242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:49:40.035843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.252:43305 (local address 127.0.0.1:51242)
0926 03:49:40.036076 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0926 03:49:40.036192 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.252:43305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260926 03:49:40.037606 23242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:49:40.037123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.252:43305 (local address 127.0.0.1:51252)
0926 03:49:40.037323 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0926 03:49:40.037435 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.252:43305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260926 03:49:40.037899 23240 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.194.252:43305: Network error: Client connection negotiation failed: client connection to 127.0.194.252:43305: connect: Connection refused (error 111)
I20260926 03:49:40.054358 23314 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.194.252" port: 43305 } attempt: 3
I20260926 03:49:40.055771 23316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:49:40.055109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.252:43305 (local address 127.0.0.1:51262)
0926 03:49:40.055384 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0926 03:49:40.055519 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.252:43305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260926 03:49:40.056986 23316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:49:40.056519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.252:43305 (local address 127.0.0.1:51278)
0926 03:49:40.056789 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0926 03:49:40.056869 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.252:43305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159}
W20260926 03:49:40.057261 23314 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.194.252:43305: Network error: Client connection negotiation failed: client connection to 127.0.194.252:43305: connect: Connection refused (error 111)
I20260926 03:49:40.106155 23240 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.194.252" port: 43305 } attempt: 2
I20260926 03:49:40.107589 23242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:49:40.106938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.252:43305 (local address 127.0.0.1:51294)
0926 03:49:40.107214 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0926 03:49:40.107351 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.252:43305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
I20260926 03:49:40.108814 23242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:49:40.108330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.252:43305 (local address 127.0.0.1:51298)
0926 03:49:40.108509 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0926 03:49:40.108618 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.252:43305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260926 03:49:40.109086 23240 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.194.252:43305: Network error: Client connection negotiation failed: client connection to 127.0.194.252:43305: connect: Connection refused (error 111)
I20260926 03:49:40.200498 23314 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.194.252" port: 43305 } attempt: 4
I20260926 03:49:40.201611 23316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:49:40.201144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.252:43305 (local address 127.0.0.1:51314)
0926 03:49:40.201380 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0926 03:49:40.201465 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.252:43305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260926 03:49:40.202636 23316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:49:40.202314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.252:43305 (local address 127.0.0.1:51320)
0926 03:49:40.202471 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0926 03:49:40.202534 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.252:43305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260926 03:49:40.202883 23314 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.194.252:43305: Network error: Client connection negotiation failed: client connection to 127.0.194.252:43305: connect: Connection refused (error 111)
W20260926 03:49:40.205188 23321 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 03:49:40.205538 23321 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:49:40.205660 23321 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:49:40.205793 23321 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:49:40.213516 23321 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:49:40.213646 23321 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:49:40.213784 23321 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:49:40.213873 23321 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:49:40.213979 23321 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:49:40.215305 23240 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.194.252" port: 43305 } attempt: 3
I20260926 03:49:40.216485 23242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:49:40.215928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.252:43305 (local address 127.0.0.1:51330)
0926 03:49:40.216175 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0926 03:49:40.216281 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.252:43305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260926 03:49:40.217614 23242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:49:40.217188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.252:43305 (local address 127.0.0.1:51346)
0926 03:49:40.217392 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0926 03:49:40.217480 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.252:43305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
W20260926 03:49:40.217831 23240 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.194.252:43305: Network error: Client connection negotiation failed: client connection to 127.0.194.252:43305: connect: Connection refused (error 111)
I20260926 03:49:40.225026 23321 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.194.212:44963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790394221564982-779-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790394221564982-779-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.194.254:45639,127.0.194.253:34749,127.0.194.252:43305
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790394221564982-779-0/krb5kdc/kudu_127.0.194.252.keytab
--principal=kudu/127.0.194.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.194.252:43305
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790394221564982-779-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.194.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790394221564982-779-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:49:40.226363 23321 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:49:40.227783 23321 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:49:40.240088 23327 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:49:40.240242 23329 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:49:40.240067 23326 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:49:40.241195 23321 server_base.cc:1034] running on GCE node
I20260926 03:49:40.242069 23321 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:49:40.243712 23321 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:49:40.244980 23321 hybrid_clock.cc:648] HybridClock initialized: now 1790394580244907 us; error 49 us; skew 500 ppm
I20260926 03:49:40.245362 23321 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:49:40 dist-test-slave-n2th krb5kdc[23160](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394580, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:49:40.248692 23321 init.cc:377] Logged in from keytab as kudu/127.0.194.252@KRBTEST.COM (short username kudu)
I20260926 03:49:40.250221 23321 webserver.cc:460] Webserver started at http://127.0.194.252:34185/ using document root <none> and password file <none>
I20260926 03:49:40.251015 23321 fs_manager.cc:359] Metadata directory not provided
I20260926 03:49:40.251153 23321 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:49:40.251494 23321 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:49:40.254554 23321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790394221564982-779-0/minicluster-data/master-2/data/instance:
uuid: "41dfc151e64144f091159bdd87f7b4ab"
format_stamp: "Formatted at 2026-09-26 03:49:40 on dist-test-slave-n2th"
server_key: "873129ff3dbfcb0b3c78a567277ccba4"
server_key_iv: "512df8348b43585767e075c00f11a290"
server_key_version: "encryptionkey@0"
I20260926 03:49:40.255430 23321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790394221564982-779-0/minicluster-data/master-2/wal/instance:
uuid: "41dfc151e64144f091159bdd87f7b4ab"
format_stamp: "Formatted at 2026-09-26 03:49:40 on dist-test-slave-n2th"
server_key: "873129ff3dbfcb0b3c78a567277ccba4"
server_key_iv: "512df8348b43585767e075c00f11a290"
server_key_version: "encryptionkey@0"
I20260926 03:49:40.260880 23321 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 03:49:40.264611 23336 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:49:40.266089 23321 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 03:49:40.266280 23321 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790394221564982-779-0/minicluster-data/master-2/data,/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790394221564982-779-0/minicluster-data/master-2/wal
uuid: "41dfc151e64144f091159bdd87f7b4ab"
format_stamp: "Formatted at 2026-09-26 03:49:40 on dist-test-slave-n2th"
server_key: "873129ff3dbfcb0b3c78a567277ccba4"
server_key_iv: "512df8348b43585767e075c00f11a290"
server_key_version: "encryptionkey@0"
I20260926 03:49:40.266510 23321 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790394221564982-779-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790394221564982-779-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790394221564982-779-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:49:40.289315 23321 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:49:40.293321 23321 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:49:40.293623 23321 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:49:40.312471 23321 rpc_server.cc:307] RPC server started. Bound to: 127.0.194.252:43305
I20260926 03:49:40.312493 23387 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.194.252:43305 every 8 connection(s)
I20260926 03:49:40.313911 23321 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790394221564982-779-0/minicluster-data/master-2/data/info.pb
I20260926 03:49:40.317836 23388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:49:40.319160 779 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu as pid 23321
I20260926 03:49:40.319360 779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790394221564982-779-0/minicluster-data/master-2/wal/instance
I20260926 03:49:40.319933 779 external_mini_cluster.cc:1428] Setting key ad1b03d51795e12116528f4d0d56e18e
I20260926 03:49:40.327983 23388 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.194.254" port: 45639 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 03:49:40 dist-test-slave-n2th krb5kdc[23160](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394578, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.194.252@KRBTEST.COM
Sep 26 03:49:40 dist-test-slave-n2th krb5kdc[23160](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394580, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.252@KRBTEST.COM for kudu/127.0.194.254@KRBTEST.COM
I20260926 03:49:40.344825 23391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:49:40.321945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51348 (local address 127.0.194.252:43305)
0926 03:49:40.322903 (+ 958us) server_negotiation.cc:206] Beginning negotiation
0926 03:49:40.322915 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 03:49:40.322960 (+ 45us) server_negotiation.cc:407] Connection header received
0926 03:49:40.324609 (+ 1649us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:49:40.324634 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:49:40.325072 (+ 438us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:49:40.325425 (+ 353us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:49:40.327414 (+ 1989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:49:40.328779 (+ 1365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:49:40.330217 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:49:40.331241 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:49:40.331471 (+ 230us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:49:40.336455 (+ 4984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:49:40.336469 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:49:40.336497 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:49:40.336546 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:49:40.339994 (+ 3448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:49:40.340842 (+ 848us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:49:40.340849 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:49:40.340856 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:49:40.341015 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:49:40.341429 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:49:40.341435 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:49:40.341438 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:49:40.342169 (+ 731us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:49:40.342508 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0926 03:49:40.342867 (+ 359us) server_negotiation.cc:299] Negotiation successful
0926 03:49:40.343352 (+ 485us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":689,"thread_start_us":444,"threads_started":1}
I20260926 03:49:40.347757 23393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:49:40.330818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:45639 (local address 127.0.0.1:37574)
0926 03:49:40.331671 (+ 853us) negotiation.cc:107] Waiting for socket to connect
0926 03:49:40.331725 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0926 03:49:40.331894 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:49:40.332674 (+ 780us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:49:40.332679 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:49:40.333311 (+ 632us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:49:40.333556 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:49:40.333571 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:49:40.334995 (+ 1424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:49:40.335002 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:49:40.336659 (+ 1657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:49:40.336671 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:49:40.337701 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:49:40.337706 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:49:40.337857 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:49:40.338493 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:49:40.338509 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:49:40.340995 (+ 2486us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:49:40.344862 (+ 3867us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:49:40.344871 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:49:40.344876 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:49:40.345459 (+ 583us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:49:40.345888 (+ 429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:49:40.345891 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:49:40.345893 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:49:40.346026 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:49:40.346615 (+ 589us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:49:40.346621 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:49:40.346824 (+ 203us) client_negotiation.cc:769] Sending connection context
0926 03:49:40.347068 (+ 244us) client_negotiation.cc:240] Negotiation successful
0926 03:49:40.347364 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":596,"thread_start_us":439,"threads_started":1}
I20260926 03:49:40.347779 23392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:49:40.330682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37574 (local address 127.0.194.254:45639)
0926 03:49:40.331254 (+ 572us) server_negotiation.cc:206] Beginning negotiation
0926 03:49:40.331259 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:49:40.331838 (+ 579us) server_negotiation.cc:407] Connection header received
0926 03:49:40.332368 (+ 530us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:49:40.332374 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:49:40.332468 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:49:40.332601 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:49:40.333734 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:49:40.334826 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:49:40.336811 (+ 1985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:49:40.337503 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:49:40.337661 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:49:40.341196 (+ 3535us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:49:40.341202 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:49:40.341207 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:49:40.341249 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:49:40.344724 (+ 3475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:49:40.345588 (+ 864us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:49:40.345594 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:49:40.345597 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:49:40.345761 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:49:40.346152 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:49:40.346157 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:49:40.346162 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:49:40.346472 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:49:40.346610 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0926 03:49:40.347147 (+ 537us) server_negotiation.cc:299] Negotiation successful
0926 03:49:40.347323 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":446,"thread_start_us":271,"threads_started":1}
I20260926 03:49:40.349303 23388 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.194.253" port: 34749 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 03:49:40 dist-test-slave-n2th krb5kdc[23160](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394580, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.252@KRBTEST.COM for kudu/127.0.194.253@KRBTEST.COM
Sep 26 03:49:40 dist-test-slave-n2th krb5kdc[23160](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394578, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.194.254@KRBTEST.COM
I20260926 03:49:40.363222 23393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:49:40.350014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.253:34749 (local address 127.0.0.1:52496)
0926 03:49:40.350207 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0926 03:49:40.350225 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 03:49:40.350324 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:49:40.350644 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:49:40.350648 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:49:40.351045 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:49:40.351193 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:49:40.351201 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:49:40.352163 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:49:40.352168 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:49:40.353027 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:49:40.353037 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:49:40.353757 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:49:40.353762 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:49:40.353951 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:49:40.354526 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:49:40.354542 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:49:40.357004 (+ 2462us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:49:40.360411 (+ 3407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:49:40.360417 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:49:40.360421 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:49:40.360859 (+ 438us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:49:40.361334 (+ 475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:49:40.361342 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:49:40.361348 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:49:40.361534 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:49:40.362169 (+ 635us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:49:40.362174 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:49:40.362342 (+ 168us) client_negotiation.cc:769] Sending connection context
0926 03:49:40.362499 (+ 157us) client_negotiation.cc:240] Negotiation successful
0926 03:49:40.362858 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66}
I20260926 03:49:40.363289 23318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:49:40.350109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52496 (local address 127.0.194.253:34749)
0926 03:49:40.350297 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0926 03:49:40.350303 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:49:40.350322 (+ 19us) server_negotiation.cc:407] Connection header received
0926 03:49:40.350430 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:49:40.350439 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:49:40.350510 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:49:40.350622 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:49:40.351348 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:49:40.352017 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:49:40.353183 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:49:40.353593 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:49:40.353729 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:49:40.357192 (+ 3463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:49:40.357197 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:49:40.357200 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:49:40.357233 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:49:40.360260 (+ 3027us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:49:40.360988 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:49:40.360996 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:49:40.361003 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:49:40.361200 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:49:40.361653 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:49:40.361660 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:49:40.361663 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:49:40.362047 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:49:40.362169 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0926 03:49:40.362619 (+ 450us) server_negotiation.cc:299] Negotiation successful
0926 03:49:40.362893 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 03:49:40.363539 23392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:49:40.348615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37578 (local address 127.0.194.254:45639)
0926 03:49:40.348804 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0926 03:49:40.348809 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:49:40.348826 (+ 17us) server_negotiation.cc:407] Connection header received
0926 03:49:40.348989 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:49:40.348995 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:49:40.349084 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:49:40.349187 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:49:40.350692 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:49:40.351423 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:49:40.352670 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:49:40.353101 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:49:40.353246 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:49:40.357870 (+ 4624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:49:40.357875 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:49:40.357878 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:49:40.357910 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:49:40.360447 (+ 2537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:49:40.361235 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:49:40.361239 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:49:40.361241 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:49:40.361463 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:49:40.361927 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:49:40.361932 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:49:40.361934 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:49:40.362292 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:49:40.362599 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0926 03:49:40.362933 (+ 334us) server_negotiation.cc:299] Negotiation successful
0926 03:49:40.363178 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 03:49:40.364696 23388 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.194.252" port: 43305 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 03:49:40 dist-test-slave-n2th krb5kdc[23160](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394580, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.252@KRBTEST.COM for kudu/127.0.194.252@KRBTEST.COM
Sep 26 03:49:40 dist-test-slave-n2th krb5kdc[23160](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394578, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.194.253@KRBTEST.COM
I20260926 03:49:40.380177 23391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:49:40.365796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51362 (local address 127.0.194.252:43305)
0926 03:49:40.365956 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0926 03:49:40.365962 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:49:40.365976 (+ 14us) server_negotiation.cc:407] Connection header received
0926 03:49:40.366025 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:49:40.366028 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:49:40.366076 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:49:40.366152 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:49:40.367138 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:49:40.368213 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:49:40.369943 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:49:40.370446 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:49:40.370599 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:49:40.374108 (+ 3509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:49:40.374114 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:49:40.374121 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:49:40.374163 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:49:40.377447 (+ 3284us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:49:40.378065 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:49:40.378071 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:49:40.378076 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:49:40.378211 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:49:40.378576 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:49:40.378580 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:49:40.378584 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:49:40.378948 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:49:40.379122 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0926 03:49:40.379437 (+ 315us) server_negotiation.cc:299] Negotiation successful
0926 03:49:40.379841 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":180,"server-negotiator.queue_time_us":59}
I20260926 03:49:40.380177 23393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:49:40.365473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.252:43305 (local address 127.0.0.1:51362)
0926 03:49:40.365649 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0926 03:49:40.365666 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 03:49:40.365763 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:49:40.366213 (+ 450us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:49:40.366220 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:49:40.366759 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:49:40.366977 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:49:40.366991 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:49:40.368340 (+ 1349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:49:40.368344 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:49:40.369773 (+ 1429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:49:40.369789 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:49:40.370762 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:49:40.370770 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:49:40.370971 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:49:40.371737 (+ 766us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:49:40.371754 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:49:40.373892 (+ 2138us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:49:40.377579 (+ 3687us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:49:40.377583 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:49:40.377585 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:49:40.377942 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:49:40.378324 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:49:40.378327 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:49:40.378329 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:49:40.378455 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:49:40.379085 (+ 630us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:49:40.379090 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:49:40.379226 (+ 136us) client_negotiation.cc:769] Sending connection context
0926 03:49:40.379366 (+ 140us) client_negotiation.cc:240] Negotiation successful
0926 03:49:40.379712 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56,"mutex_wait_us":3}
I20260926 03:49:40.381560 23318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:49:40.365273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52506 (local address 127.0.194.253:34749)
0926 03:49:40.365432 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0926 03:49:40.365439 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:49:40.365461 (+ 22us) server_negotiation.cc:407] Connection header received
0926 03:49:40.365662 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:49:40.365667 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:49:40.365739 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:49:40.365833 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:49:40.367401 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:49:40.368415 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:49:40.370059 (+ 1644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:49:40.370537 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:49:40.370681 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:49:40.375234 (+ 4553us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:49:40.375241 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:49:40.375246 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:49:40.375284 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:49:40.378341 (+ 3057us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:49:40.379066 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:49:40.379072 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:49:40.379075 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:49:40.379223 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:49:40.379602 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:49:40.379606 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:49:40.379610 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:49:40.379866 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:49:40.380147 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0926 03:49:40.381030 (+ 883us) server_negotiation.cc:299] Negotiation successful
0926 03:49:40.381277 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 03:49:40.386035 23388 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41dfc151e64144f091159bdd87f7b4ab: Bootstrap starting.
I20260926 03:49:40.386679 779 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 03:49:40.389536 23388 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 41dfc151e64144f091159bdd87f7b4ab: Neither blocks nor log segments found. Creating new log.
I20260926 03:49:40.390663 23388 log.cc:824] T 00000000000000000000000000000000 P 41dfc151e64144f091159bdd87f7b4ab: Log is configured to *not* fsync() on all Append() calls
I20260926 03:49:40.389400 23215 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "9139e44175f74472bddc1f3e3c159754"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37578
I20260926 03:49:40.393647 23388 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41dfc151e64144f091159bdd87f7b4ab: No bootstrap required, opened a new log
I20260926 03:49:40.395028 23240 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.194.252" port: 43305 } attempt: 4
I20260926 03:49:40.403594 23388 raft_consensus.cc:348] T 00000000000000000000000000000000 P 41dfc151e64144f091159bdd87f7b4ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9139e44175f74472bddc1f3e3c159754" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 45639 } } peers { permanent_uuid: "20b4675e77d1407bba747dca53ba150e" member_type: VOTER last_known_addr { host: "127.0.194.253" port: 34749 } } peers { permanent_uuid: "41dfc151e64144f091159bdd87f7b4ab" member_type: VOTER last_known_addr { host: "127.0.194.252" port: 43305 } }
I20260926 03:49:40.404083 23388 raft_consensus.cc:729] T 00000000000000000000000000000000 P 41dfc151e64144f091159bdd87f7b4ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41dfc151e64144f091159bdd87f7b4ab, State: Initialized, Role: FOLLOWER
Sep 26 03:49:40 dist-test-slave-n2th krb5kdc[23160](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394579, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.254@KRBTEST.COM for kudu/127.0.194.252@KRBTEST.COM
I20260926 03:49:40.404567 23388 consensus_queue.cc:260] T 00000000000000000000000000000000 P 41dfc151e64144f091159bdd87f7b4ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9139e44175f74472bddc1f3e3c159754" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 45639 } } peers { permanent_uuid: "20b4675e77d1407bba747dca53ba150e" member_type: VOTER last_known_addr { host: "127.0.194.253" port: 34749 } } peers { permanent_uuid: "41dfc151e64144f091159bdd87f7b4ab" member_type: VOTER last_known_addr { host: "127.0.194.252" port: 43305 } }
I20260926 03:49:40.405822 23395 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41dfc151e64144f091159bdd87f7b4ab [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9139e44175f74472bddc1f3e3c159754" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 45639 } } peers { permanent_uuid: "20b4675e77d1407bba747dca53ba150e" member_type: VOTER last_known_addr { host: "127.0.194.253" port: 34749 } } peers { permanent_uuid: "41dfc151e64144f091159bdd87f7b4ab" member_type: VOTER last_known_addr { host: "127.0.194.252" port: 43305 } } }
I20260926 03:49:40.406221 23395 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41dfc151e64144f091159bdd87f7b4ab [sys.catalog]: This master's current role is: FOLLOWER
I20260926 03:49:40.407106 23388 sys_catalog.cc:564] T 00000000000000000000000000000000 P 41dfc151e64144f091159bdd87f7b4ab [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:49:40.412933 23242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:49:40.395711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.252:43305 (local address 127.0.0.1:51364)
0926 03:49:40.395924 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0926 03:49:40.395947 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 03:49:40.396077 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:49:40.396544 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:49:40.396549 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:49:40.397060 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:49:40.397284 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:49:40.397295 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:49:40.398881 (+ 1586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:49:40.398887 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:49:40.400457 (+ 1570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:49:40.400473 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:49:40.401619 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:49:40.401627 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:49:40.401859 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:49:40.402605 (+ 746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:49:40.402630 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:49:40.405397 (+ 2767us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:49:40.410119 (+ 4722us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:49:40.410124 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:49:40.410127 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:49:40.410544 (+ 417us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:49:40.411059 (+ 515us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:49:40.411071 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:49:40.411074 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:49:40.411218 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:49:40.411943 (+ 725us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:49:40.411951 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:49:40.412162 (+ 211us) client_negotiation.cc:769] Sending connection context
0926 03:49:40.412355 (+ 193us) client_negotiation.cc:240] Negotiation successful
0926 03:49:40.412602 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67}
I20260926 03:49:40.414119 23391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:49:40.395785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51364 (local address 127.0.194.252:43305)
0926 03:49:40.395978 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0926 03:49:40.395982 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:49:40.396003 (+ 21us) server_negotiation.cc:407] Connection header received
0926 03:49:40.396233 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:49:40.396238 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:49:40.396357 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:49:40.396491 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:49:40.397440 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:49:40.398659 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:49:40.400660 (+ 2001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:49:40.401443 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:49:40.401575 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:49:40.405639 (+ 4064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:49:40.405645 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:49:40.405650 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:49:40.405689 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:49:40.409937 (+ 4248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:49:40.410699 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:49:40.410706 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:49:40.410710 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:49:40.410922 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:49:40.411340 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:49:40.411349 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:49:40.411354 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:49:40.411789 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:49:40.412009 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0926 03:49:40.413473 (+ 1464us) server_negotiation.cc:299] Negotiation successful
0926 03:49:40.413724 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
W20260926 03:49:40.417843 23406 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 41dfc151e64144f091159bdd87f7b4ab: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 03:49:40.417972 23406 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 03:49:40.419916 23240 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9139e44175f74472bddc1f3e3c159754: Bootstrap starting.
I20260926 03:49:40.424134 23240 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9139e44175f74472bddc1f3e3c159754: Neither blocks nor log segments found. Creating new log.
I20260926 03:49:40.425498 23240 log.cc:824] T 00000000000000000000000000000000 P 9139e44175f74472bddc1f3e3c159754: Log is configured to *not* fsync() on all Append() calls
I20260926 03:49:40.418867 23321 master_runner.cc:186] Error getting master registration for 127.0.194.254:45639: OK, instance_id { permanent_uuid: "9139e44175f74472bddc1f3e3c159754" instance_seqno: 1790394579253603 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 03:49:40.431900 23321 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 03:49:40.435372 23240 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9139e44175f74472bddc1f3e3c159754: No bootstrap required, opened a new log
I20260926 03:49:40.440917 23240 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9139e44175f74472bddc1f3e3c159754 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9139e44175f74472bddc1f3e3c159754" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 45639 } } peers { permanent_uuid: "20b4675e77d1407bba747dca53ba150e" member_type: VOTER last_known_addr { host: "127.0.194.253" port: 34749 } } peers { permanent_uuid: "41dfc151e64144f091159bdd87f7b4ab" member_type: VOTER last_known_addr { host: "127.0.194.252" port: 43305 } }
I20260926 03:49:40.441320 23240 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9139e44175f74472bddc1f3e3c159754 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9139e44175f74472bddc1f3e3c159754, State: Initialized, Role: FOLLOWER
I20260926 03:49:40.441864 23240 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9139e44175f74472bddc1f3e3c159754 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9139e44175f74472bddc1f3e3c159754" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 45639 } } peers { permanent_uuid: "20b4675e77d1407bba747dca53ba150e" member_type: VOTER last_known_addr { host: "127.0.194.253" port: 34749 } } peers { permanent_uuid: "41dfc151e64144f091159bdd87f7b4ab" member_type: VOTER last_known_addr { host: "127.0.194.252" port: 43305 } }
I20260926 03:49:40.442880 23409 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9139e44175f74472bddc1f3e3c159754 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9139e44175f74472bddc1f3e3c159754" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 45639 } } peers { permanent_uuid: "20b4675e77d1407bba747dca53ba150e" member_type: VOTER last_known_addr { host: "127.0.194.253" port: 34749 } } peers { permanent_uuid: "41dfc151e64144f091159bdd87f7b4ab" member_type: VOTER last_known_addr { host: "127.0.194.252" port: 43305 } } }
I20260926 03:49:40.443256 23409 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9139e44175f74472bddc1f3e3c159754 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 03:49:40.444070 23240 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9139e44175f74472bddc1f3e3c159754 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:49:40.444204 23215 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9139e44175f74472bddc1f3e3c159754 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260926 03:49:40.444468 23215 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9139e44175f74472bddc1f3e3c159754 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:49:40.448307 23215 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9139e44175f74472bddc1f3e3c159754 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9139e44175f74472bddc1f3e3c159754" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 45639 } } peers { permanent_uuid: "20b4675e77d1407bba747dca53ba150e" member_type: VOTER last_known_addr { host: "127.0.194.253" port: 34749 } } peers { permanent_uuid: "41dfc151e64144f091159bdd87f7b4ab" member_type: VOTER last_known_addr { host: "127.0.194.252" port: 43305 } }
I20260926 03:49:40.451138 23363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9139e44175f74472bddc1f3e3c159754" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "41dfc151e64144f091159bdd87f7b4ab"
I20260926 03:49:40.451742 23363 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 41dfc151e64144f091159bdd87f7b4ab [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:49:40.454955 23215 leader_election.cc:288] T 00000000000000000000000000000000 P 9139e44175f74472bddc1f3e3c159754 [CANDIDATE]: Term 1 election: Requested vote from peers 20b4675e77d1407bba747dca53ba150e (127.0.194.253:34749), 41dfc151e64144f091159bdd87f7b4ab (127.0.194.252:43305)
I20260926 03:49:40.455685 23363 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 41dfc151e64144f091159bdd87f7b4ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9139e44175f74472bddc1f3e3c159754 in term 1.
I20260926 03:49:40.456409 23191 leader_election.cc:302] T 00000000000000000000000000000000 P 9139e44175f74472bddc1f3e3c159754 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 41dfc151e64144f091159bdd87f7b4ab, 9139e44175f74472bddc1f3e3c159754; no voters:
I20260926 03:49:40.458810 23409 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9139e44175f74472bddc1f3e3c159754 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:49:40.478545 23409 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9139e44175f74472bddc1f3e3c159754 [term 1 LEADER]: Becoming Leader. State: Replica: 9139e44175f74472bddc1f3e3c159754, State: Running, Role: LEADER
I20260926 03:49:40.479351 23409 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9139e44175f74472bddc1f3e3c159754 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9139e44175f74472bddc1f3e3c159754" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 45639 } } peers { permanent_uuid: "20b4675e77d1407bba747dca53ba150e" member_type: VOTER last_known_addr { host: "127.0.194.253" port: 34749 } } peers { permanent_uuid: "41dfc151e64144f091159bdd87f7b4ab" member_type: VOTER last_known_addr { host: "127.0.194.252" port: 43305 } }
I20260926 03:49:40.486220 23418 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9139e44175f74472bddc1f3e3c159754 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9139e44175f74472bddc1f3e3c159754. Latest consensus state: current_term: 1 leader_uuid: "9139e44175f74472bddc1f3e3c159754" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9139e44175f74472bddc1f3e3c159754" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 45639 } } peers { permanent_uuid: "20b4675e77d1407bba747dca53ba150e" member_type: VOTER last_known_addr { host: "127.0.194.253" port: 34749 } } peers { permanent_uuid: "41dfc151e64144f091159bdd87f7b4ab" member_type: VOTER last_known_addr { host: "127.0.194.252" port: 43305 } } }
I20260926 03:49:40.486831 23418 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9139e44175f74472bddc1f3e3c159754 [sys.catalog]: This master's current role is: LEADER
W20260926 03:49:40.495110 23194 tablet.cc:2369] T 00000000000000000000000000000000 P 9139e44175f74472bddc1f3e3c159754: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 03:49:40.500496 23427 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9139e44175f74472bddc1f3e3c159754: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 03:49:40.500784 23427 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 03:49:40.502074 23314 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.194.252" port: 43305 } attempt: 5
I20260926 03:49:40.526721 23428 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:49:40.530134 23392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:49:40.484052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37588 (local address 127.0.194.254:45639)
0926 03:49:40.484257 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0926 03:49:40.484264 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:49:40.487710 (+ 3446us) server_negotiation.cc:407] Connection header received
0926 03:49:40.488002 (+ 292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:49:40.488009 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:49:40.488095 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:49:40.488162 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:49:40.505397 (+ 17235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:49:40.506971 (+ 1574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:49:40.509367 (+ 2396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:49:40.510206 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:49:40.510352 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:49:40.515685 (+ 5333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:49:40.515696 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:49:40.515700 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:49:40.515752 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:49:40.520091 (+ 4339us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:49:40.521039 (+ 948us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:49:40.521045 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:49:40.521049 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:49:40.521217 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:49:40.521738 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:49:40.521743 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:49:40.521747 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:49:40.522187 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:49:40.529207 (+ 7020us) server_negotiation.cc:1035] Waiting for connection context
0926 03:49:40.529393 (+ 186us) server_negotiation.cc:299] Negotiation successful
0926 03:49:40.529689 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 03:49:40.544528 23428 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260926 03:49:40.545843 23342 tablet.cc:2369] T 00000000000000000000000000000000 P 41dfc151e64144f091159bdd87f7b4ab: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 03:49:40.490777 23173 master_runner.cc:186] Error getting master registration for 127.0.194.253:34749: OK, instance_id { permanent_uuid: "20b4675e77d1407bba747dca53ba150e" instance_seqno: 1790394579771179 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 03:49:40.547547 23173 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 03:49:40.557036 23318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:49:40.484958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52520 (local address 127.0.194.253:34749)
0926 03:49:40.485174 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0926 03:49:40.485182 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:49:40.523552 (+ 38370us) server_negotiation.cc:407] Connection header received
0926 03:49:40.523826 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:49:40.523833 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:49:40.523920 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:49:40.523995 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:49:40.525958 (+ 1963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:49:40.527146 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:49:40.542886 (+ 15740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:49:40.543909 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:49:40.544022 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:49:40.547850 (+ 3828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:49:40.547857 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:49:40.547861 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:49:40.547919 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:49:40.551629 (+ 3710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:49:40.552606 (+ 977us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:49:40.552614 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:49:40.552617 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:49:40.552792 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:49:40.553262 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:49:40.553267 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:49:40.553271 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:49:40.553679 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:49:40.553935 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0926 03:49:40.556324 (+ 2389us) server_negotiation.cc:299] Negotiation successful
0926 03:49:40.556610 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":292,"server-negotiator.queue_time_us":46}
Sep 26 03:49:40 dist-test-slave-n2th krb5kdc[23160](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394579, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.253@KRBTEST.COM for kudu/127.0.194.252@KRBTEST.COM
I20260926 03:49:40.563869 23363 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 41dfc151e64144f091159bdd87f7b4ab [term 1 FOLLOWER]: Refusing update from remote peer 9139e44175f74472bddc1f3e3c159754: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:49:40.565121 23418 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9139e44175f74472bddc1f3e3c159754 [LEADER]: Connected to new peer: Peer: permanent_uuid: "41dfc151e64144f091159bdd87f7b4ab" member_type: VOTER last_known_addr { host: "127.0.194.252" port: 43305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:49:40.583854 23316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:49:40.502761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.252:43305 (local address 127.0.0.1:51382)
0926 03:49:40.503012 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0926 03:49:40.503033 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 03:49:40.503162 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:49:40.536258 (+ 33096us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:49:40.536265 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:49:40.536804 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:49:40.537028 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:49:40.537043 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:49:40.542137 (+ 5094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:49:40.542144 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:49:40.543694 (+ 1550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:49:40.543714 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:49:40.559783 (+ 16069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:49:40.559790 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:49:40.560084 (+ 294us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:49:40.560803 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:49:40.560830 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:49:40.563766 (+ 2936us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:49:40.578478 (+ 14712us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:49:40.578485 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:49:40.578489 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:49:40.579161 (+ 672us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:49:40.579763 (+ 602us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:49:40.579770 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:49:40.579775 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:49:40.579978 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:49:40.582656 (+ 2678us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:49:40.582662 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:49:40.582899 (+ 237us) client_negotiation.cc:769] Sending connection context
0926 03:49:40.583054 (+ 155us) client_negotiation.cc:240] Negotiation successful
0926 03:49:40.583406 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"mutex_wait_us":1}
I20260926 03:49:40.585500 23431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:49:40.503232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51382 (local address 127.0.194.252:43305)
0926 03:49:40.532965 (+ 29733us) server_negotiation.cc:206] Beginning negotiation
0926 03:49:40.532971 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:49:40.532997 (+ 26us) server_negotiation.cc:407] Connection header received
0926 03:49:40.533062 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:49:40.533069 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:49:40.533153 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:49:40.537197 (+ 4044us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:49:40.537503 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:49:40.538861 (+ 1358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:49:40.558812 (+ 19951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:49:40.559582 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:49:40.563968 (+ 4386us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:49:40.564246 (+ 278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:49:40.564253 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:49:40.564258 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:49:40.564296 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:49:40.573108 (+ 8812us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:49:40.579346 (+ 6238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:49:40.579353 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:49:40.579359 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:49:40.579598 (+ 239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:49:40.580144 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:49:40.580148 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:49:40.580153 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:49:40.580589 (+ 436us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:49:40.584613 (+ 4024us) server_negotiation.cc:1035] Waiting for connection context
0926 03:49:40.584763 (+ 150us) server_negotiation.cc:299] Negotiation successful
0926 03:49:40.585022 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29586,"thread_start_us":29445,"threads_started":1}
I20260926 03:49:40.586681 23391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:49:40.496802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51368 (local address 127.0.194.252:43305)
0926 03:49:40.497012 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0926 03:49:40.497019 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:49:40.531507 (+ 34488us) server_negotiation.cc:407] Connection header received
0926 03:49:40.533741 (+ 2234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:49:40.533749 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:49:40.533821 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:49:40.533891 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:49:40.539051 (+ 5160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:49:40.540149 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:49:40.565393 (+ 25244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:49:40.566134 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:49:40.566257 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:49:40.573369 (+ 7112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:49:40.573377 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:49:40.573382 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:49:40.573425 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:49:40.577381 (+ 3956us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:49:40.580744 (+ 3363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:49:40.580750 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:49:40.580756 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:49:40.580896 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:49:40.581425 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:49:40.581433 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:49:40.581440 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:49:40.581824 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:49:40.582087 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0926 03:49:40.586027 (+ 3940us) server_negotiation.cc:299] Negotiation successful
0926 03:49:40.586301 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260926 03:49:40.590634 23395 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41dfc151e64144f091159bdd87f7b4ab [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9139e44175f74472bddc1f3e3c159754. Latest consensus state: current_term: 1 leader_uuid: "9139e44175f74472bddc1f3e3c159754" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9139e44175f74472bddc1f3e3c159754" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 45639 } } peers { permanent_uuid: "20b4675e77d1407bba747dca53ba150e" member_type: VOTER last_known_addr { host: "127.0.194.253" port: 34749 } } peers { permanent_uuid: "41dfc151e64144f091159bdd87f7b4ab" member_type: VOTER last_known_addr { host: "127.0.194.252" port: 43305 } } }
I20260926 03:49:40.592782 23395 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41dfc151e64144f091159bdd87f7b4ab [sys.catalog]: This master's current role is: FOLLOWER
I20260926 03:49:40.596022 23314 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20b4675e77d1407bba747dca53ba150e: Bootstrap starting.
I20260926 03:49:40.596706 23433 mvcc.cc:205] Tried to move back new op lower bound from 7333456201974456320 to 7333456201649278976. Current Snapshot: MvccSnapshot[applied={T|T < 7333456201974456320}]
I20260926 03:49:40.601619 23314 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 20b4675e77d1407bba747dca53ba150e: Neither blocks nor log segments found. Creating new log.
I20260926 03:49:40.602507 23418 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9139e44175f74472bddc1f3e3c159754 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9139e44175f74472bddc1f3e3c159754" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9139e44175f74472bddc1f3e3c159754" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 45639 } } peers { permanent_uuid: "20b4675e77d1407bba747dca53ba150e" member_type: VOTER last_known_addr { host: "127.0.194.253" port: 34749 } } peers { permanent_uuid: "41dfc151e64144f091159bdd87f7b4ab" member_type: VOTER last_known_addr { host: "127.0.194.252" port: 43305 } } }
I20260926 03:49:40.602897 23418 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9139e44175f74472bddc1f3e3c159754 [sys.catalog]: This master's current role is: LEADER
I20260926 03:49:40.604091 23434 mvcc.cc:205] Tried to move back new op lower bound from 7333456201974456320 to 7333456201649278976. Current Snapshot: MvccSnapshot[applied={T|T < 7333456201974456320}]
I20260926 03:49:40.604218 23314 log.cc:824] T 00000000000000000000000000000000 P 20b4675e77d1407bba747dca53ba150e: Log is configured to *not* fsync() on all Append() calls
I20260926 03:49:40.611078 23428 catalog_manager.cc:1347] Generated new cluster ID: 0515ecd8424c412a8a4b212d9cc900c7
I20260926 03:49:40.611222 23428 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:49:40.615221 23395 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41dfc151e64144f091159bdd87f7b4ab [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9139e44175f74472bddc1f3e3c159754" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9139e44175f74472bddc1f3e3c159754" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 45639 } } peers { permanent_uuid: "20b4675e77d1407bba747dca53ba150e" member_type: VOTER last_known_addr { host: "127.0.194.253" port: 34749 } } peers { permanent_uuid: "41dfc151e64144f091159bdd87f7b4ab" member_type: VOTER last_known_addr { host: "127.0.194.252" port: 43305 } } }
I20260926 03:49:40.615532 23395 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41dfc151e64144f091159bdd87f7b4ab [sys.catalog]: This master's current role is: FOLLOWER
I20260926 03:49:40.617583 23314 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20b4675e77d1407bba747dca53ba150e: No bootstrap required, opened a new log
I20260926 03:49:40.633162 23428 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:49:40.635964 23314 raft_consensus.cc:348] T 00000000000000000000000000000000 P 20b4675e77d1407bba747dca53ba150e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9139e44175f74472bddc1f3e3c159754" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 45639 } } peers { permanent_uuid: "20b4675e77d1407bba747dca53ba150e" member_type: VOTER last_known_addr { host: "127.0.194.253" port: 34749 } } peers { permanent_uuid: "41dfc151e64144f091159bdd87f7b4ab" member_type: VOTER last_known_addr { host: "127.0.194.252" port: 43305 } }
I20260926 03:49:40.636809 23314 raft_consensus.cc:729] T 00000000000000000000000000000000 P 20b4675e77d1407bba747dca53ba150e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20b4675e77d1407bba747dca53ba150e, State: Initialized, Role: FOLLOWER
I20260926 03:49:40.637742 23314 consensus_queue.cc:260] T 00000000000000000000000000000000 P 20b4675e77d1407bba747dca53ba150e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9139e44175f74472bddc1f3e3c159754" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 45639 } } peers { permanent_uuid: "20b4675e77d1407bba747dca53ba150e" member_type: VOTER last_known_addr { host: "127.0.194.253" port: 34749 } } peers { permanent_uuid: "41dfc151e64144f091159bdd87f7b4ab" member_type: VOTER last_known_addr { host: "127.0.194.252" port: 43305 } }
I20260926 03:49:40.639408 23439 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20b4675e77d1407bba747dca53ba150e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9139e44175f74472bddc1f3e3c159754" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 45639 } } peers { permanent_uuid: "20b4675e77d1407bba747dca53ba150e" member_type: VOTER last_known_addr { host: "127.0.194.253" port: 34749 } } peers { permanent_uuid: "41dfc151e64144f091159bdd87f7b4ab" member_type: VOTER last_known_addr { host: "127.0.194.252" port: 43305 } } }
I20260926 03:49:40.640232 23439 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20b4675e77d1407bba747dca53ba150e [sys.catalog]: This master's current role is: FOLLOWER
I20260926 03:49:40.640640 23314 sys_catalog.cc:564] T 00000000000000000000000000000000 P 20b4675e77d1407bba747dca53ba150e [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:49:40.640726 23289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9139e44175f74472bddc1f3e3c159754" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "20b4675e77d1407bba747dca53ba150e"
I20260926 03:49:40.641191 23289 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 20b4675e77d1407bba747dca53ba150e [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:49:40.649528 23289 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 20b4675e77d1407bba747dca53ba150e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9139e44175f74472bddc1f3e3c159754 in term 1.
I20260926 03:49:40.650408 23288 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 20b4675e77d1407bba747dca53ba150e [term 1 FOLLOWER]: Refusing update from remote peer 9139e44175f74472bddc1f3e3c159754: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:49:40.651206 23428 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:49:40.652374 23409 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9139e44175f74472bddc1f3e3c159754 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20b4675e77d1407bba747dca53ba150e" member_type: VOTER last_known_addr { host: "127.0.194.253" port: 34749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:49:40.692180 23428 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9139e44175f74472bddc1f3e3c159754: Generated new TSK 0
I20260926 03:49:40.693454 23428 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 03:49:40.701452 23418 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9139e44175f74472bddc1f3e3c159754 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9139e44175f74472bddc1f3e3c159754" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9139e44175f74472bddc1f3e3c159754" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 45639 } } peers { permanent_uuid: "20b4675e77d1407bba747dca53ba150e" member_type: VOTER last_known_addr { host: "127.0.194.253" port: 34749 } } peers { permanent_uuid: "41dfc151e64144f091159bdd87f7b4ab" member_type: VOTER last_known_addr { host: "127.0.194.252" port: 43305 } } }
I20260926 03:49:40.701749 23418 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9139e44175f74472bddc1f3e3c159754 [sys.catalog]: This master's current role is: LEADER
I20260926 03:49:40.702831 23439 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20b4675e77d1407bba747dca53ba150e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9139e44175f74472bddc1f3e3c159754. Latest consensus state: current_term: 1 leader_uuid: "9139e44175f74472bddc1f3e3c159754" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9139e44175f74472bddc1f3e3c159754" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 45639 } } peers { permanent_uuid: "20b4675e77d1407bba747dca53ba150e" member_type: VOTER last_known_addr { host: "127.0.194.253" port: 34749 } } peers { permanent_uuid: "41dfc151e64144f091159bdd87f7b4ab" member_type: VOTER last_known_addr { host: "127.0.194.252" port: 43305 } } }
I20260926 03:49:40.703207 23439 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20b4675e77d1407bba747dca53ba150e [sys.catalog]: This master's current role is: FOLLOWER
I20260926 03:49:40.703539 23447 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20b4675e77d1407bba747dca53ba150e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9139e44175f74472bddc1f3e3c159754" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9139e44175f74472bddc1f3e3c159754" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 45639 } } peers { permanent_uuid: "20b4675e77d1407bba747dca53ba150e" member_type: VOTER last_known_addr { host: "127.0.194.253" port: 34749 } } peers { permanent_uuid: "41dfc151e64144f091159bdd87f7b4ab" member_type: VOTER last_known_addr { host: "127.0.194.252" port: 43305 } } }
I20260926 03:49:40.703914 23447 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20b4675e77d1407bba747dca53ba150e [sys.catalog]: This master's current role is: FOLLOWER
I20260926 03:49:40.751082 23456 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
W20260926 03:49:40.789410 23431 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:49:40.776375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51396 (local address 127.0.194.252:43305)
0926 03:49:40.776576 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0926 03:49:40.776586 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 03:49:40.787426 (+ 10840us) server_negotiation.cc:407] Connection header received
0926 03:49:40.787731 (+ 305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:49:40.787737 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:49:40.787846 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:49:40.787916 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:49:40.789168 (+ 1252us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51396: BlockingRecv error: recv got EOF from 127.0.0.1:51396 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
W20260926 03:49:40.792603 23318 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:49:40.776976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52524 (local address 127.0.194.253:34749)
0926 03:49:40.777209 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0926 03:49:40.777214 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:49:40.790426 (+ 13212us) server_negotiation.cc:407] Connection header received
0926 03:49:40.791034 (+ 608us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:49:40.791040 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:49:40.791123 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:49:40.791194 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:49:40.792345 (+ 1151us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52524: BlockingRecv error: recv got EOF from 127.0.0.1:52524 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260926 03:49:40.795279 23392 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:49:40.788197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37596 (local address 127.0.194.254:45639)
0926 03:49:40.788432 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0926 03:49:40.788437 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:49:40.793334 (+ 4897us) server_negotiation.cc:407] Connection header received
0926 03:49:40.793502 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:49:40.793511 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:49:40.793624 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:49:40.793761 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:49:40.795014 (+ 1253us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37596: BlockingRecv error: recv got EOF from 127.0.0.1:37596 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260926 03:49:40.815629 23318 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:49:40.813177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52540 (local address 127.0.194.253:34749)
0926 03:49:40.813383 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0926 03:49:40.813389 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:49:40.813953 (+ 564us) server_negotiation.cc:407] Connection header received
0926 03:49:40.814213 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:49:40.814219 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:49:40.814348 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:49:40.814465 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:49:40.815342 (+ 877us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52540: BlockingRecv error: recv got EOF from 127.0.0.1:52540 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
W20260926 03:49:40.827521 23318 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:49:40.825175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52548 (local address 127.0.194.253:34749)
0926 03:49:40.825359 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0926 03:49:40.825365 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:49:40.825911 (+ 546us) server_negotiation.cc:407] Connection header received
0926 03:49:40.826120 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:49:40.826126 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:49:40.826226 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:49:40.826339 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:49:40.827292 (+ 953us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52548: BlockingRecv error: recv got EOF from 127.0.0.1:52548 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260926 03:49:40.840721 23318 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:49:40.838341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52556 (local address 127.0.194.253:34749)
0926 03:49:40.838539 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0926 03:49:40.838545 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:49:40.839112 (+ 567us) server_negotiation.cc:407] Connection header received
0926 03:49:40.839337 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:49:40.839344 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:49:40.839453 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:49:40.839573 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:49:40.840468 (+ 895us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52556: BlockingRecv error: recv got EOF from 127.0.0.1:52556 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
W20260926 03:49:40.850948 23318 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:49:40.848529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52564 (local address 127.0.194.253:34749)
0926 03:49:40.848760 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0926 03:49:40.848766 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:49:40.849339 (+ 573us) server_negotiation.cc:407] Connection header received
0926 03:49:40.849548 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:49:40.849555 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:49:40.849670 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:49:40.849785 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:49:40.850666 (+ 881us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52564: BlockingRecv error: recv got EOF from 127.0.0.1:52564 (error 108)
Metrics: {"server-negotiator.queue_time_us":106}
W20260926 03:49:40.867375 23318 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:49:40.864896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52566 (local address 127.0.194.253:34749)
0926 03:49:40.865095 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0926 03:49:40.865101 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:49:40.865696 (+ 595us) server_negotiation.cc:407] Connection header received
0926 03:49:40.865914 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:49:40.865921 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:49:40.866031 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:49:40.866142 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:49:40.867128 (+ 986us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52566: BlockingRecv error: recv got EOF from 127.0.0.1:52566 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260926 03:49:40.895218 23318 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:49:40.892717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52574 (local address 127.0.194.253:34749)
0926 03:49:40.892913 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0926 03:49:40.892931 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0926 03:49:40.893494 (+ 563us) server_negotiation.cc:407] Connection header received
0926 03:49:40.893705 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:49:40.893711 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:49:40.893828 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:49:40.893954 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:49:40.894960 (+ 1006us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52574: BlockingRecv error: recv got EOF from 127.0.0.1:52574 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
W20260926 03:49:40.935213 23318 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:49:40.932719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52578 (local address 127.0.194.253:34749)
0926 03:49:40.932939 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0926 03:49:40.932946 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:49:40.933534 (+ 588us) server_negotiation.cc:407] Connection header received
0926 03:49:40.933749 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:49:40.933756 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:49:40.933876 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:49:40.934001 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:49:40.934946 (+ 945us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52578: BlockingRecv error: recv got EOF from 127.0.0.1:52578 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
W20260926 03:49:41.006620 23318 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:49:41.004166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52588 (local address 127.0.194.253:34749)
0926 03:49:41.004364 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0926 03:49:41.004373 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:49:41.005006 (+ 633us) server_negotiation.cc:407] Connection header received
0926 03:49:41.005205 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:49:41.005211 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:49:41.005312 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:49:41.005428 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:49:41.006371 (+ 943us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52588: BlockingRecv error: recv got EOF from 127.0.0.1:52588 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
W20260926 03:49:41.140856 23318 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:49:41.138709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52590 (local address 127.0.194.253:34749)
0926 03:49:41.138983 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0926 03:49:41.138991 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:49:41.139350 (+ 359us) server_negotiation.cc:407] Connection header received
0926 03:49:41.139585 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:49:41.139591 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:49:41.139682 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:49:41.139798 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:49:41.140583 (+ 785us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52590: BlockingRecv error: recv got EOF from 127.0.0.1:52590 (error 108)
Metrics: {"server-negotiator.queue_time_us":100}
W20260926 03:49:41.406812 23318 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:49:41.404890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52606 (local address 127.0.194.253:34749)
0926 03:49:41.405088 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0926 03:49:41.405092 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:49:41.405440 (+ 348us) server_negotiation.cc:407] Connection header received
0926 03:49:41.405636 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:49:41.405640 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:49:41.405713 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:49:41.405808 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:49:41.406562 (+ 754us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52606: BlockingRecv error: recv got EOF from 127.0.0.1:52606 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 03:49:41.420056 23406 catalog_manager.cc:1259] Loaded cluster ID: 0515ecd8424c412a8a4b212d9cc900c7
I20260926 03:49:41.420164 23406 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 41dfc151e64144f091159bdd87f7b4ab: loading cluster ID for follower catalog manager: success
I20260926 03:49:41.422659 23406 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 41dfc151e64144f091159bdd87f7b4ab: acquiring CA information for follower catalog manager: success
I20260926 03:49:41.424539 23406 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 41dfc151e64144f091159bdd87f7b4ab: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 03:49:41.755565 23456 catalog_manager.cc:1259] Loaded cluster ID: 0515ecd8424c412a8a4b212d9cc900c7
I20260926 03:49:41.755686 23456 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 20b4675e77d1407bba747dca53ba150e: loading cluster ID for follower catalog manager: success
I20260926 03:49:41.758605 23456 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 20b4675e77d1407bba747dca53ba150e: acquiring CA information for follower catalog manager: success
I20260926 03:49:41.760550 23456 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 20b4675e77d1407bba747dca53ba150e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 03:49:41.943814 23520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:49:41.936465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52620 (local address 127.0.194.253:34749)
0926 03:49:41.937072 (+ 607us) server_negotiation.cc:206] Beginning negotiation
0926 03:49:41.937081 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:49:41.937388 (+ 307us) server_negotiation.cc:407] Connection header received
0926 03:49:41.937638 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:49:41.937642 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:49:41.937784 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:49:41.937918 (+ 134us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:49:41.938404 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:49:41.939449 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:49:41.940969 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:49:41.941626 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:49:41.941791 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:49:41.942379 (+ 588us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:49:41.943071 (+ 692us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:49:41.943243 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0926 03:49:41.943372 (+ 129us) server_negotiation.cc:299] Negotiation successful
0926 03:49:41.943468 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":469,"thread_start_us":301,"threads_started":1}
I20260926 03:49:41.958586 23527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:49:41.951642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51404 (local address 127.0.194.252:43305)
0926 03:49:41.952202 (+ 560us) server_negotiation.cc:206] Beginning negotiation
0926 03:49:41.952208 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:49:41.952569 (+ 361us) server_negotiation.cc:407] Connection header received
0926 03:49:41.952808 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:49:41.952812 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:49:41.952961 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:49:41.953127 (+ 166us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:49:41.953584 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:49:41.954290 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:49:41.956238 (+ 1948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:49:41.956655 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:49:41.956752 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:49:41.957382 (+ 630us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:49:41.957918 (+ 536us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:49:41.958027 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0926 03:49:41.958187 (+ 160us) server_negotiation.cc:299] Negotiation successful
0926 03:49:41.958262 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":398,"thread_start_us":266,"threads_started":1}
I20260926 03:49:41.980645 23527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:49:41.965851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51406 (local address 127.0.194.252:43305)
0926 03:49:41.966065 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0926 03:49:41.966074 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:49:41.973358 (+ 7284us) server_negotiation.cc:407] Connection header received
0926 03:49:41.973692 (+ 334us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:49:41.973699 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:49:41.973809 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:49:41.973881 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:49:41.974660 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:49:41.975846 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:49:41.978274 (+ 2428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:49:41.978939 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:49:41.979044 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:49:41.979743 (+ 699us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:49:41.979892 (+ 149us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:49:41.979958 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 03:49:41.980225 (+ 267us) server_negotiation.cc:299] Negotiation successful
0926 03:49:41.980319 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260926 03:49:41.983659 23520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:49:41.965492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52622 (local address 127.0.194.253:34749)
0926 03:49:41.965739 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0926 03:49:41.965745 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:49:41.976021 (+ 10276us) server_negotiation.cc:407] Connection header received
0926 03:49:41.976539 (+ 518us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:49:41.976545 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:49:41.976639 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:49:41.976709 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:49:41.977696 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:49:41.978982 (+ 1286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:49:41.981371 (+ 2389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:49:41.982024 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:49:41.982153 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:49:41.982795 (+ 642us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:49:41.982938 (+ 143us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:49:41.983012 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 03:49:41.983252 (+ 240us) server_negotiation.cc:299] Negotiation successful
0926 03:49:41.983337 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260926 03:49:41.984644 23533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:49:41.965591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37598 (local address 127.0.194.254:45639)
0926 03:49:41.975514 (+ 9923us) server_negotiation.cc:206] Beginning negotiation
0926 03:49:41.975522 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:49:41.976472 (+ 950us) server_negotiation.cc:407] Connection header received
0926 03:49:41.976609 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:49:41.976614 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:49:41.976847 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:49:41.977021 (+ 174us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:49:41.978027 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:49:41.979165 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:49:41.981764 (+ 2599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:49:41.982374 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:49:41.982508 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:49:41.983282 (+ 774us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:49:41.983743 (+ 461us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:49:41.983852 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0926 03:49:41.984332 (+ 480us) server_negotiation.cc:299] Negotiation successful
0926 03:49:41.984424 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9779,"thread_start_us":9679,"threads_started":1}
I20260926 03:49:41.989609 779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu pid 23173
I20260926 03:49:41.990899 23235 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:49:42.109724 779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu with pid 23173
I20260926 03:49:42.138829 779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu pid 23247
I20260926 03:49:42.139981 23309 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:49:42.249992 779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu with pid 23247
I20260926 03:49:42.270206 779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu pid 23321
I20260926 03:49:42.271392 23383 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:49:42.381217 779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu with pid 23321
2026-09-26T03:49:42Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (4965 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790394221564982-779-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:49:42 dist-test-slave-n2th krb5kdc[23541](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:49:42 dist-test-slave-n2th krb5kdc[23541](info): set up 2 sockets
Sep 26 03:49:42 dist-test-slave-n2th krb5kdc[23541](info): commencing operation
krb5kdc: starting...
W20260926 03:49:43.519381 779 mini_kdc.cc:121] Time spent starting KDC: real 1.096s user 0.002s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790394221564982-779-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790394221564982-779-0/krb5kdc/test-user.keytab.
Sep 26 03:49:43 dist-test-slave-n2th krb5kdc[23541](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394583, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:49:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:49:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.194.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.194.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.194.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790394221564982-779-0/krb5kdc/kudu_127.0.194.254.keytab.
Entry for principal kudu/127.0.194.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790394221564982-779-0/krb5kdc/kudu_127.0.194.254.keytab.
I20260926 03:49:43.797534 779 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
/tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790394221564982-779-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790394221564982-779-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790394221564982-779-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790394221564982-779-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.194.254:38429
--webserver_interface=127.0.194.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790394221564982-779-0/krb5kdc/kudu_127.0.194.254.keytab
--principal=kudu/127.0.194.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.194.212:38689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.194.254:38429,127.0.194.253:43771,127.0.194.252:40195
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790394221564982-779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790394221564982-779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790394221564982-779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:49:44.141991 23554 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 03:49:44.142335 23554 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:49:44.142409 23554 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:49:44.142493 23554 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:49:44.150015 23554 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:49:44.150122 23554 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:49:44.150185 23554 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:49:44.150231 23554 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:49:44.150295 23554 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:49:44.160651 23554 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.194.212:38689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790394221564982-779-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790394221564982-779-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.194.254:38429,127.0.194.253:43771,127.0.194.252:40195
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790394221564982-779-0/krb5kdc/kudu_127.0.194.254.keytab
--principal=kudu/127.0.194.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.194.254:38429
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790394221564982-779-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.194.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790394221564982-779-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:49:44.161978 23554 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:49:44.163411 23554 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:49:44.175864 23562 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:49:44.175901 23560 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:49:44.175942 23559 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:49:44.176838 23554 server_base.cc:1034] running on GCE node
I20260926 03:49:44.177731 23554 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:49:44.179345 23554 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:49:44.180600 23554 hybrid_clock.cc:648] HybridClock initialized: now 1790394584180526 us; error 38 us; skew 500 ppm
I20260926 03:49:44.180998 23554 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:49:44 dist-test-slave-n2th krb5kdc[23541](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394584, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:49:44.184525 23554 init.cc:377] Logged in from keytab as kudu/127.0.194.254@KRBTEST.COM (short username kudu)
I20260926 03:49:44.186112 23554 webserver.cc:460] Webserver started at http://127.0.194.254:39565/ using document root <none> and password file <none>
I20260926 03:49:44.186929 23554 fs_manager.cc:359] Metadata directory not provided
I20260926 03:49:44.187055 23554 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:49:44.187424 23554 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:49:44.190546 23554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790394221564982-779-0/minicluster-data/master-0/data/instance:
uuid: "ae8444917e8a49959886c1bfd249dd95"
format_stamp: "Formatted at 2026-09-26 03:49:44 on dist-test-slave-n2th"
server_key: "5f8d0e8a969b9dcabb5a56d9d550c824"
server_key_iv: "5bedaf69cc307fc16b1c6402abe82553"
server_key_version: "encryptionkey@0"
I20260926 03:49:44.191459 23554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790394221564982-779-0/minicluster-data/master-0/wal/instance:
uuid: "ae8444917e8a49959886c1bfd249dd95"
format_stamp: "Formatted at 2026-09-26 03:49:44 on dist-test-slave-n2th"
server_key: "5f8d0e8a969b9dcabb5a56d9d550c824"
server_key_iv: "5bedaf69cc307fc16b1c6402abe82553"
server_key_version: "encryptionkey@0"
I20260926 03:49:44.196884 23554 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 03:49:44.200512 23569 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:49:44.201857 23554 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 03:49:44.202051 23554 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790394221564982-779-0/minicluster-data/master-0/data,/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790394221564982-779-0/minicluster-data/master-0/wal
uuid: "ae8444917e8a49959886c1bfd249dd95"
format_stamp: "Formatted at 2026-09-26 03:49:44 on dist-test-slave-n2th"
server_key: "5f8d0e8a969b9dcabb5a56d9d550c824"
server_key_iv: "5bedaf69cc307fc16b1c6402abe82553"
server_key_version: "encryptionkey@0"
I20260926 03:49:44.202296 23554 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790394221564982-779-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790394221564982-779-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790394221564982-779-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:49:44.229604 23554 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:49:44.233804 23554 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:49:44.234117 23554 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:49:44.253669 23554 rpc_server.cc:307] RPC server started. Bound to: 127.0.194.254:38429
I20260926 03:49:44.253690 23620 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.194.254:38429 every 8 connection(s)
I20260926 03:49:44.255281 23554 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790394221564982-779-0/minicluster-data/master-0/data/info.pb
I20260926 03:49:44.259124 23621 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:49:44.264554 779 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu as pid 23554
I20260926 03:49:44.264750 779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790394221564982-779-0/minicluster-data/master-0/wal/instance
I20260926 03:49:44.265271 779 external_mini_cluster.cc:1428] Setting key 75a724a0bcb1b7e091707cf3ff7ae20e
I20260926 03:49:44.266829 23621 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.194.254" port: 38429 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 03:49:44 dist-test-slave-n2th krb5kdc[23541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394584, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.254@KRBTEST.COM for kudu/127.0.194.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.194.253@KRBTEST.COM; defaulting to no policy
I20260926 03:49:44.291069 23623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:49:44.271201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:38429 (local address 127.0.0.1:49432)
0926 03:49:44.272132 (+ 931us) negotiation.cc:107] Waiting for socket to connect
0926 03:49:44.272179 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 03:49:44.274259 (+ 2080us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:49:44.275262 (+ 1003us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:49:44.275269 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:49:44.276006 (+ 737us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:49:44.276343 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:49:44.276364 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:49:44.278001 (+ 1637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:49:44.278007 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:49:44.279201 (+ 1194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:49:44.279214 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:49:44.280150 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:49:44.280156 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:49:44.280324 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:49:44.281034 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:49:44.281057 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:49:44.283322 (+ 2265us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:49:44.286522 (+ 3200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:49:44.286535 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:49:44.286549 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:49:44.287055 (+ 506us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:49:44.287475 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:49:44.287482 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:49:44.287487 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:49:44.287652 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:49:44.288458 (+ 806us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:49:44.288467 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:49:44.288695 (+ 228us) client_negotiation.cc:769] Sending connection context
0926 03:49:44.288936 (+ 241us) client_negotiation.cc:240] Negotiation successful
0926 03:49:44.289620 (+ 684us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":655,"mutex_wait_us":195,"thread_start_us":423,"threads_started":1}
I20260926 03:49:44.291111 23624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:49:44.271201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49432 (local address 127.0.194.254:38429)
0926 03:49:44.272991 (+ 1790us) server_negotiation.cc:206] Beginning negotiation
0926 03:49:44.272999 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:49:44.273049 (+ 50us) server_negotiation.cc:407] Connection header received
0926 03:49:44.274670 (+ 1621us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:49:44.274677 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:49:44.275021 (+ 344us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:49:44.275238 (+ 217us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:49:44.276567 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:49:44.277839 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:49:44.279338 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:49:44.279903 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:49:44.280125 (+ 222us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:49:44.283553 (+ 3428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:49:44.283561 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:49:44.283574 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:49:44.283605 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:49:44.286377 (+ 2772us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:49:44.287203 (+ 826us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:49:44.287210 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:49:44.287212 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:49:44.287361 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:49:44.287766 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:49:44.287777 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:49:44.287779 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:49:44.288320 (+ 541us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:49:44.288531 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0926 03:49:44.289068 (+ 537us) server_negotiation.cc:299] Negotiation successful
0926 03:49:44.289487 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":40,"server-negotiator.queue_time_us":1566,"thread_start_us":1337,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.194.253@KRBTEST.COM" created.
I20260926 03:49:44.294448 23621 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.194.253" port: 43771 } has no permanent_uuid. Determining permanent_uuid...
I20260926 03:49:44.295776 23623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:49:44.295218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.253:43771 (local address 127.0.0.1:55486)
0926 03:49:44.295451 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0926 03:49:44.295592 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.253:43771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260926 03:49:44.296074 23573 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.194.253:43771: connect: Connection refused (error 111)
I20260926 03:49:44.297770 23623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:49:44.297323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.253:43771 (local address 127.0.0.1:55498)
0926 03:49:44.297493 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0926 03:49:44.297590 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.253:43771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260926 03:49:44.298084 23621 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.194.253:43771: Network error: Client connection negotiation failed: client connection to 127.0.194.253:43771: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.194.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790394221564982-779-0/krb5kdc/kudu_127.0.194.253.keytab.
Entry for principal kudu/127.0.194.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790394221564982-779-0/krb5kdc/kudu_127.0.194.253.keytab.
I20260926 03:49:44.325553 779 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
/tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790394221564982-779-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790394221564982-779-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790394221564982-779-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790394221564982-779-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.194.253:43771
--webserver_interface=127.0.194.253
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790394221564982-779-0/krb5kdc/kudu_127.0.194.253.keytab
--principal=kudu/127.0.194.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.194.212:38689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.194.254:38429,127.0.194.253:43771,127.0.194.252:40195
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790394221564982-779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790394221564982-779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790394221564982-779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 03:49:44.347417 23621 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.194.253" port: 43771 } attempt: 1
I20260926 03:49:44.348677 23623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:49:44.348224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.253:43771 (local address 127.0.0.1:55510)
0926 03:49:44.348431 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0926 03:49:44.348547 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.253:43771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260926 03:49:44.349682 23623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:49:44.349319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.253:43771 (local address 127.0.0.1:55512)
0926 03:49:44.349460 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0926 03:49:44.349531 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.253:43771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260926 03:49:44.349869 23621 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.194.253:43771: Network error: Client connection negotiation failed: client connection to 127.0.194.253:43771: connect: Connection refused (error 111)
I20260926 03:49:44.418147 23621 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.194.253" port: 43771 } attempt: 2
I20260926 03:49:44.419374 23623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:49:44.418928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.253:43771 (local address 127.0.0.1:55522)
0926 03:49:44.419141 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0926 03:49:44.419232 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.253:43771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260926 03:49:44.420374 23623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:49:44.420019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.253:43771 (local address 127.0.0.1:55526)
0926 03:49:44.420165 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0926 03:49:44.420250 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.253:43771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260926 03:49:44.420559 23621 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.194.253:43771: Network error: Client connection negotiation failed: client connection to 127.0.194.253:43771: connect: Connection refused (error 111)
I20260926 03:49:44.511796 23621 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.194.253" port: 43771 } attempt: 3
I20260926 03:49:44.512979 23623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:49:44.512522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.253:43771 (local address 127.0.0.1:55534)
0926 03:49:44.512732 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0926 03:49:44.512830 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.253:43771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260926 03:49:44.513967 23623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:49:44.513611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.253:43771 (local address 127.0.0.1:55548)
0926 03:49:44.513752 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0926 03:49:44.513827 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.253:43771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260926 03:49:44.514138 23621 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.194.253:43771: Network error: Client connection negotiation failed: client connection to 127.0.194.253:43771: connect: Connection refused (error 111)
I20260926 03:49:44.657407 23621 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.194.253" port: 43771 } attempt: 4
I20260926 03:49:44.658715 23623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:49:44.658165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.253:43771 (local address 127.0.0.1:55550)
0926 03:49:44.658488 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0926 03:49:44.658577 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.253:43771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171}
I20260926 03:49:44.659905 23623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:49:44.659562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.253:43771 (local address 127.0.0.1:55560)
0926 03:49:44.659702 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0926 03:49:44.659788 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.253:43771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260926 03:49:44.660082 23621 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.194.253:43771: Network error: Client connection negotiation failed: client connection to 127.0.194.253:43771: connect: Connection refused (error 111)
W20260926 03:49:44.667511 23628 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 03:49:44.667917 23628 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:49:44.668049 23628 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:49:44.668186 23628 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:49:44.675787 23628 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:49:44.675932 23628 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:49:44.676064 23628 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:49:44.676160 23628 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:49:44.676277 23628 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:49:44.686844 23628 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.194.212:38689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790394221564982-779-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790394221564982-779-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.194.254:38429,127.0.194.253:43771,127.0.194.252:40195
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790394221564982-779-0/krb5kdc/kudu_127.0.194.253.keytab
--principal=kudu/127.0.194.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.194.253:43771
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790394221564982-779-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.194.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790394221564982-779-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:49:44.688272 23628 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:49:44.689642 23628 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:49:44.702414 23634 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:49:44.702375 23633 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:49:44.702844 23636 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:49:44.703634 23628 server_base.cc:1034] running on GCE node
I20260926 03:49:44.704584 23628 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:49:44.706240 23628 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:49:44.707540 23628 hybrid_clock.cc:648] HybridClock initialized: now 1790394584707455 us; error 69 us; skew 500 ppm
I20260926 03:49:44.707855 23628 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:49:44 dist-test-slave-n2th krb5kdc[23541](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394584, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:49:44.710951 23628 init.cc:377] Logged in from keytab as kudu/127.0.194.253@KRBTEST.COM (short username kudu)
I20260926 03:49:44.712549 23628 webserver.cc:460] Webserver started at http://127.0.194.253:39065/ using document root <none> and password file <none>
I20260926 03:49:44.713258 23628 fs_manager.cc:359] Metadata directory not provided
I20260926 03:49:44.713388 23628 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:49:44.713706 23628 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:49:44.716774 23628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790394221564982-779-0/minicluster-data/master-1/data/instance:
uuid: "35ccc0996edd48bd9c5ff73e1264394e"
format_stamp: "Formatted at 2026-09-26 03:49:44 on dist-test-slave-n2th"
server_key: "1e76e390debc4c3ac62a42fd94921f66"
server_key_iv: "dd141dfa36b0d2fb31a402ddb81dec54"
server_key_version: "encryptionkey@0"
I20260926 03:49:44.717592 23628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790394221564982-779-0/minicluster-data/master-1/wal/instance:
uuid: "35ccc0996edd48bd9c5ff73e1264394e"
format_stamp: "Formatted at 2026-09-26 03:49:44 on dist-test-slave-n2th"
server_key: "1e76e390debc4c3ac62a42fd94921f66"
server_key_iv: "dd141dfa36b0d2fb31a402ddb81dec54"
server_key_version: "encryptionkey@0"
I20260926 03:49:44.723203 23628 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260926 03:49:44.726853 23643 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:49:44.728307 23628 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 03:49:44.728508 23628 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790394221564982-779-0/minicluster-data/master-1/data,/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790394221564982-779-0/minicluster-data/master-1/wal
uuid: "35ccc0996edd48bd9c5ff73e1264394e"
format_stamp: "Formatted at 2026-09-26 03:49:44 on dist-test-slave-n2th"
server_key: "1e76e390debc4c3ac62a42fd94921f66"
server_key_iv: "dd141dfa36b0d2fb31a402ddb81dec54"
server_key_version: "encryptionkey@0"
I20260926 03:49:44.728740 23628 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790394221564982-779-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790394221564982-779-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790394221564982-779-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:49:44.754124 23628 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:49:44.758324 23628 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:49:44.758610 23628 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:49:44.777513 23628 rpc_server.cc:307] RPC server started. Bound to: 127.0.194.253:43771
I20260926 03:49:44.777527 23694 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.194.253:43771 every 8 connection(s)
I20260926 03:49:44.778980 23628 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790394221564982-779-0/minicluster-data/master-1/data/info.pb
I20260926 03:49:44.783011 779 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu as pid 23628
I20260926 03:49:44.783154 23695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:49:44.783224 779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790394221564982-779-0/minicluster-data/master-1/wal/instance
I20260926 03:49:44.783792 779 external_mini_cluster.cc:1428] Setting key 345cc9baf4966610ec0068d7beb8354c
I20260926 03:49:44.793650 23695 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.194.254" port: 38429 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 03:49:44 dist-test-slave-n2th krb5kdc[23541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394584, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.253@KRBTEST.COM for kudu/127.0.194.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.194.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.194.252@KRBTEST.COM" created.
I20260926 03:49:44.816469 23698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:49:44.796899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49446 (local address 127.0.194.254:38429)
0926 03:49:44.797427 (+ 528us) server_negotiation.cc:206] Beginning negotiation
0926 03:49:44.797436 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:49:44.798497 (+ 1061us) server_negotiation.cc:407] Connection header received
0926 03:49:44.800468 (+ 1971us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:49:44.800478 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:49:44.800573 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:49:44.800702 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:49:44.802259 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:49:44.803505 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:49:44.805019 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:49:44.805814 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:49:44.805967 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:49:44.809747 (+ 3780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:49:44.809754 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:49:44.809760 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:49:44.809822 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:49:44.813140 (+ 3318us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:49:44.814017 (+ 877us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:49:44.814023 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:49:44.814027 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:49:44.814165 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:49:44.814591 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:49:44.814598 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:49:44.814601 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:49:44.814965 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:49:44.815123 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0926 03:49:44.815861 (+ 738us) server_negotiation.cc:299] Negotiation successful
0926 03:49:44.816091 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":249,"threads_started":1}
I20260926 03:49:44.817562 23699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:49:44.797481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:38429 (local address 127.0.0.1:49446)
0926 03:49:44.798306 (+ 825us) negotiation.cc:107] Waiting for socket to connect
0926 03:49:44.798356 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 03:49:44.800194 (+ 1838us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:49:44.800901 (+ 707us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:49:44.800909 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:49:44.801477 (+ 568us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:49:44.802038 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:49:44.802055 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:49:44.803660 (+ 1605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:49:44.803663 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:49:44.804869 (+ 1206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:49:44.804881 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:49:44.806068 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:49:44.806074 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:49:44.806350 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:49:44.807136 (+ 786us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:49:44.807161 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:49:44.809490 (+ 2329us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:49:44.813283 (+ 3793us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:49:44.813290 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:49:44.813303 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:49:44.813871 (+ 568us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:49:44.814283 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:49:44.814289 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:49:44.814297 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:49:44.814476 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:49:44.815096 (+ 620us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:49:44.815104 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:49:44.815490 (+ 386us) client_negotiation.cc:769] Sending connection context
0926 03:49:44.815783 (+ 293us) client_negotiation.cc:240] Negotiation successful
0926 03:49:44.816336 (+ 553us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":558,"thread_start_us":347,"threads_started":1}
I20260926 03:49:44.819334 23695 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.194.253" port: 43771 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 03:49:44 dist-test-slave-n2th krb5kdc[23541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394584, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.253@KRBTEST.COM for kudu/127.0.194.253@KRBTEST.COM
I20260926 03:49:44.836938 23700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:49:44.820504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55564 (local address 127.0.194.253:43771)
0926 03:49:44.821277 (+ 773us) server_negotiation.cc:206] Beginning negotiation
0926 03:49:44.821287 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 03:49:44.821322 (+ 35us) server_negotiation.cc:407] Connection header received
0926 03:49:44.821381 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:49:44.821391 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:49:44.821658 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:49:44.821816 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:49:44.822780 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:49:44.823856 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:49:44.825754 (+ 1898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:49:44.826283 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:49:44.826377 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:49:44.830499 (+ 4122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:49:44.830512 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:49:44.830528 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:49:44.830576 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:49:44.833746 (+ 3170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:49:44.834608 (+ 862us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:49:44.834615 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:49:44.834617 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:49:44.834770 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:49:44.835194 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:49:44.835197 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:49:44.835199 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:49:44.835547 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:49:44.835809 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0926 03:49:44.836258 (+ 449us) server_negotiation.cc:299] Negotiation successful
0926 03:49:44.836616 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":138,"server-negotiator.queue_time_us":575,"thread_start_us":416,"threads_started":1}
I20260926 03:49:44.837028 23699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:49:44.820271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.253:43771 (local address 127.0.0.1:55564)
0926 03:49:44.820478 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0926 03:49:44.820498 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 03:49:44.820613 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:49:44.821843 (+ 1230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:49:44.821850 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:49:44.822367 (+ 517us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:49:44.822570 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:49:44.822581 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:49:44.824001 (+ 1420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:49:44.824010 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:49:44.825606 (+ 1596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:49:44.825623 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:49:44.826441 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:49:44.826449 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:49:44.826686 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:49:44.827392 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:49:44.827419 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:49:44.830260 (+ 2841us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:49:44.833884 (+ 3624us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:49:44.833892 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:49:44.833900 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:49:44.834468 (+ 568us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:49:44.834888 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:49:44.834896 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:49:44.834903 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:49:44.835085 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:49:44.835774 (+ 689us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:49:44.835783 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:49:44.835968 (+ 185us) client_negotiation.cc:769] Sending connection context
0926 03:49:44.836131 (+ 163us) client_negotiation.cc:240] Negotiation successful
0926 03:49:44.836485 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68}
I20260926 03:49:44.839718 23695 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.194.252" port: 40195 } has no permanent_uuid. Determining permanent_uuid...
I20260926 03:49:44.840912 23699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:49:44.840393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.252:40195 (local address 127.0.0.1:34642)
0926 03:49:44.840560 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0926 03:49:44.840754 (+ 194us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.252:40195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260926 03:49:44.841179 23647 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.194.252:40195: connect: Connection refused (error 111)
I20260926 03:49:44.842911 23699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:49:44.842512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.252:40195 (local address 127.0.0.1:34658)
0926 03:49:44.842671 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0926 03:49:44.842800 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.252:40195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260926 03:49:44.843134 23695 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.194.252:40195: Network error: Client connection negotiation failed: client connection to 127.0.194.252:40195: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.194.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790394221564982-779-0/krb5kdc/kudu_127.0.194.252.keytab.
Entry for principal kudu/127.0.194.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790394221564982-779-0/krb5kdc/kudu_127.0.194.252.keytab.
I20260926 03:49:44.848031 779 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
/tmp/dist-test-tasktGRWtf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790394221564982-779-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790394221564982-779-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790394221564982-779-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790394221564982-779-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.194.252:40195
--webserver_interface=127.0.194.252
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790394221564982-779-0/krb5kdc/kudu_127.0.194.252.keytab
--principal=kudu/127.0.194.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.194.212:38689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.194.254:38429,127.0.194.253:43771,127.0.194.252:40195
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790394221564982-779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790394221564982-779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790394221564982-779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 03:49:44.892506 23695 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.194.252" port: 40195 } attempt: 1
I20260926 03:49:44.893847 23699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:49:44.893261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.252:40195 (local address 127.0.0.1:34660)
0926 03:49:44.893494 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0926 03:49:44.893627 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.252:40195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260926 03:49:44.895095 23699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:49:44.894579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.252:40195 (local address 127.0.0.1:34672)
0926 03:49:44.894784 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0926 03:49:44.894921 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.252:40195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260926 03:49:44.895381 23695 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.194.252:40195: Network error: Client connection negotiation failed: client connection to 127.0.194.252:40195: connect: Connection refused (error 111)
I20260926 03:49:44.959367 23621 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.194.253" port: 43771 } attempt: 5
I20260926 03:49:44.963651 23695 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.194.252" port: 40195 } attempt: 2
I20260926 03:49:44.964790 23699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:49:44.964203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.252:40195 (local address 127.0.0.1:34680)
0926 03:49:44.964413 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0926 03:49:44.964521 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.252:40195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260926 03:49:44.965840 23699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:49:44.965466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.252:40195 (local address 127.0.0.1:34692)
0926 03:49:44.965612 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0926 03:49:44.965698 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.252:40195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260926 03:49:44.966986 23695 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.194.252:40195: Network error: Client connection negotiation failed: client connection to 127.0.194.252:40195: connect: Connection refused (error 111)
Sep 26 03:49:44 dist-test-slave-n2th krb5kdc[23541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394584, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.254@KRBTEST.COM for kudu/127.0.194.253@KRBTEST.COM
I20260926 03:49:44.973798 23623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:49:44.960243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.253:43771 (local address 127.0.0.1:55568)
0926 03:49:44.960481 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0926 03:49:44.960505 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 03:49:44.960682 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:49:44.961074 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:49:44.961079 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:49:44.961493 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:49:44.961678 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:49:44.961687 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:49:44.962663 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:49:44.962670 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:49:44.963614 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:49:44.963626 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:49:44.964729 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:49:44.964739 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:49:44.965010 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:49:44.965755 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:49:44.965778 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:49:44.967960 (+ 2182us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:49:44.971607 (+ 3647us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:49:44.971611 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:49:44.971614 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:49:44.972007 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:49:44.972387 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:49:44.972390 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:49:44.972392 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:49:44.972517 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:49:44.973112 (+ 595us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:49:44.973115 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:49:44.973234 (+ 119us) client_negotiation.cc:769] Sending connection context
0926 03:49:44.973341 (+ 107us) client_negotiation.cc:240] Negotiation successful
0926 03:49:44.973559 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":82}
I20260926 03:49:44.973978 23700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:49:44.960282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55568 (local address 127.0.194.253:43771)
0926 03:49:44.960499 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0926 03:49:44.960507 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:49:44.960600 (+ 93us) server_negotiation.cc:407] Connection header received
0926 03:49:44.960825 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:49:44.960831 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:49:44.960903 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:49:44.961038 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:49:44.961832 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:49:44.962528 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:49:44.963767 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:49:44.964484 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:49:44.964619 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:49:44.968187 (+ 3568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:49:44.968193 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:49:44.968197 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:49:44.968238 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:49:44.971476 (+ 3238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:49:44.972125 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:49:44.972130 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:49:44.972134 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:49:44.972282 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:49:44.972622 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:49:44.972626 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:49:44.972629 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:49:44.973005 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:49:44.973148 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0926 03:49:44.973476 (+ 328us) server_negotiation.cc:299] Negotiation successful
0926 03:49:44.973698 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260926 03:49:44.975212 23621 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.194.252" port: 40195 } has no permanent_uuid. Determining permanent_uuid...
I20260926 03:49:44.976225 23623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:49:44.975859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.252:40195 (local address 127.0.0.1:34700)
0926 03:49:44.976013 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0926 03:49:44.976095 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.252:40195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260926 03:49:44.977219 23623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:49:44.976916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.252:40195 (local address 127.0.0.1:34716)
0926 03:49:44.977046 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0926 03:49:44.977131 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.252:40195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260926 03:49:44.977506 23621 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.194.252:40195: Network error: Client connection negotiation failed: client connection to 127.0.194.252:40195: connect: Connection refused (error 111)
I20260926 03:49:45.028728 23621 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.194.252" port: 40195 } attempt: 1
I20260926 03:49:45.030170 23623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:49:45.029470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.252:40195 (local address 127.0.0.1:34732)
0926 03:49:45.029778 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0926 03:49:45.029916 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.252:40195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
I20260926 03:49:45.031462 23623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:49:45.030986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.252:40195 (local address 127.0.0.1:34736)
0926 03:49:45.031182 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0926 03:49:45.031294 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.252:40195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260926 03:49:45.031723 23621 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.194.252:40195: Network error: Client connection negotiation failed: client connection to 127.0.194.252:40195: connect: Connection refused (error 111)
I20260926 03:49:45.058267 23695 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.194.252" port: 40195 } attempt: 3
I20260926 03:49:45.059533 23699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:49:45.058969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.252:40195 (local address 127.0.0.1:34742)
0926 03:49:45.059215 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0926 03:49:45.059330 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.252:40195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260926 03:49:45.060706 23699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:49:45.060216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.252:40195 (local address 127.0.0.1:34746)
0926 03:49:45.060457 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0926 03:49:45.060552 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.252:40195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127}
W20260926 03:49:45.060923 23695 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.194.252:40195: Network error: Client connection negotiation failed: client connection to 127.0.194.252:40195: connect: Connection refused (error 111)
I20260926 03:49:45.099941 23621 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.194.252" port: 40195 } attempt: 2
I20260926 03:49:45.101358 23623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:49:45.100840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.252:40195 (local address 127.0.0.1:34754)
0926 03:49:45.101100 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0926 03:49:45.101195 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.252:40195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260926 03:49:45.102361 23623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:49:45.102024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.252:40195 (local address 127.0.0.1:34764)
0926 03:49:45.102177 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0926 03:49:45.102268 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.252:40195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260926 03:49:45.102584 23621 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.194.252:40195: Network error: Client connection negotiation failed: client connection to 127.0.194.252:40195: connect: Connection refused (error 111)
W20260926 03:49:45.195073 23703 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 03:49:45.195473 23703 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:49:45.195600 23703 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:49:45.195740 23703 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:49:45.203233 23703 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:49:45.203364 23703 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:49:45.203485 23703 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:49:45.203572 23703 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:49:45.203679 23703 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:49:45.204130 23695 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.194.252" port: 40195 } attempt: 4
I20260926 03:49:45.205297 23699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:49:45.204765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.252:40195 (local address 127.0.0.1:34772)
0926 03:49:45.205004 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0926 03:49:45.205112 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.252:40195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260926 03:49:45.206326 23699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:49:45.205975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.252:40195 (local address 127.0.0.1:34780)
0926 03:49:45.206101 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0926 03:49:45.206179 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.252:40195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260926 03:49:45.206552 23695 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.194.252:40195: Network error: Client connection negotiation failed: client connection to 127.0.194.252:40195: connect: Connection refused (error 111)
I20260926 03:49:45.208771 23621 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.194.252" port: 40195 } attempt: 3
I20260926 03:49:45.209712 23623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:49:45.209328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.252:40195 (local address 127.0.0.1:34782)
0926 03:49:45.209486 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0926 03:49:45.209576 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.252:40195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260926 03:49:45.210621 23623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:49:45.210324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.252:40195 (local address 127.0.0.1:34788)
0926 03:49:45.210462 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0926 03:49:45.210521 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.194.252:40195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260926 03:49:45.210925 23621 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.194.252:40195: Network error: Client connection negotiation failed: client connection to 127.0.194.252:40195: connect: Connection refused (error 111)
I20260926 03:49:45.215284 23703 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.194.212:38689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790394221564982-779-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790394221564982-779-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.194.254:38429,127.0.194.253:43771,127.0.194.252:40195
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790394221564982-779-0/krb5kdc/kudu_127.0.194.252.keytab
--principal=kudu/127.0.194.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.194.252:40195
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790394221564982-779-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.194.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790394221564982-779-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:49:45.216609 23703 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:49:45.217926 23703 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:49:45.229701 23708 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:49:45.229663 23711 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:49:45.229748 23709 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:49:45.232152 23703 server_base.cc:1034] running on GCE node
I20260926 03:49:45.233187 23703 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:49:45.234715 23703 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:49:45.236040 23703 hybrid_clock.cc:648] HybridClock initialized: now 1790394585235953 us; error 33 us; skew 500 ppm
I20260926 03:49:45.236438 23703 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:49:45 dist-test-slave-n2th krb5kdc[23541](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394585, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:49:45.239593 23703 init.cc:377] Logged in from keytab as kudu/127.0.194.252@KRBTEST.COM (short username kudu)
I20260926 03:49:45.241194 23703 webserver.cc:460] Webserver started at http://127.0.194.252:35259/ using document root <none> and password file <none>
I20260926 03:49:45.241961 23703 fs_manager.cc:359] Metadata directory not provided
I20260926 03:49:45.242105 23703 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:49:45.242429 23703 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:49:45.245595 23703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790394221564982-779-0/minicluster-data/master-2/data/instance:
uuid: "945ec0d86aab48f58802ba5ecd6fb0a4"
format_stamp: "Formatted at 2026-09-26 03:49:45 on dist-test-slave-n2th"
server_key: "1bcbadbfb1a979ca3105d4cdf16ea73d"
server_key_iv: "6c657886fb7333dd7e9a44ec3093abc4"
server_key_version: "encryptionkey@0"
I20260926 03:49:45.246416 23703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790394221564982-779-0/minicluster-data/master-2/wal/instance:
uuid: "945ec0d86aab48f58802ba5ecd6fb0a4"
format_stamp: "Formatted at 2026-09-26 03:49:45 on dist-test-slave-n2th"
server_key: "1bcbadbfb1a979ca3105d4cdf16ea73d"
server_key_iv: "6c657886fb7333dd7e9a44ec3093abc4"
server_key_version: "encryptionkey@0"
I20260926 03:49:45.251807 23703 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260926 03:49:45.255944 23718 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:49:45.257627 23703 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.002s
I20260926 03:49:45.257824 23703 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790394221564982-779-0/minicluster-data/master-2/data,/tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790394221564982-779-0/minicluster-data/master-2/wal
uuid: "945ec0d86aab48f58802ba5ecd6fb0a4"
format_stamp: "Formatted at 2026-09-26 03:49:45 on dist-test-slave-n2th"
server_key: "1bcbadbfb1a979ca3105d4cdf16ea73d"
server_key_iv: "6c657886fb7333dd7e9a44ec3093abc4"
server_key_version: "encryptionkey@0"
I20260926 03:49:45.258083 23703 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790394221564982-779-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790394221564982-779-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790394221564982-779-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:49:45.277980 23703 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:49:45.282069 23703 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:49:45.282295 23703 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:49:45.300830 23703 rpc_server.cc:307] RPC server started. Bound to: 127.0.194.252:40195
I20260926 03:49:45.300837 23769 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.194.252:40195 every 8 connection(s)
I20260926 03:49:45.302242 23703 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790394221564982-779-0/minicluster-data/master-2/data/info.pb
I20260926 03:49:45.305743 779 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu as pid 23703
I20260926 03:49:45.305958 779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktGRWtf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790394221564982-779-0/minicluster-data/master-2/wal/instance
I20260926 03:49:45.306108 23770 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:49:45.306505 779 external_mini_cluster.cc:1428] Setting key 31e187959b8353e01b2ffee7db448d17
I20260926 03:49:45.316450 23770 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.194.254" port: 38429 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 03:49:45 dist-test-slave-n2th krb5kdc[23541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394583, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.194.252@KRBTEST.COM
Sep 26 03:49:45 dist-test-slave-n2th krb5kdc[23541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394585, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.252@KRBTEST.COM for kudu/127.0.194.254@KRBTEST.COM
I20260926 03:49:45.331291 23773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:49:45.308413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34802 (local address 127.0.194.252:40195)
0926 03:49:45.309339 (+ 926us) server_negotiation.cc:206] Beginning negotiation
0926 03:49:45.309355 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 03:49:45.309409 (+ 54us) server_negotiation.cc:407] Connection header received
0926 03:49:45.311356 (+ 1947us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:49:45.311387 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:49:45.311942 (+ 555us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:49:45.312407 (+ 465us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:49:45.313938 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:49:45.315875 (+ 1937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:49:45.317272 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:49:45.317959 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:49:45.318167 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:49:45.323210 (+ 5043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:49:45.323218 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:49:45.323229 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:49:45.323270 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:49:45.326992 (+ 3722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:49:45.327839 (+ 847us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:49:45.327846 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:49:45.327853 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:49:45.328063 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:49:45.328478 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:49:45.328481 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:49:45.328483 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:49:45.328996 (+ 513us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:49:45.329315 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0926 03:49:45.329764 (+ 449us) server_negotiation.cc:299] Negotiation successful
0926 03:49:45.330192 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":665,"thread_start_us":353,"threads_started":1}
I20260926 03:49:45.335122 23774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:49:45.318898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.254:38429 (local address 127.0.0.1:49462)
0926 03:49:45.319824 (+ 926us) negotiation.cc:107] Waiting for socket to connect
0926 03:49:45.319880 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0926 03:49:45.320073 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:49:45.320794 (+ 721us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:49:45.320799 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:49:45.321374 (+ 575us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:49:45.321614 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:49:45.321622 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:49:45.322857 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:49:45.322863 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:49:45.324120 (+ 1257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:49:45.324132 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:49:45.324996 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:49:45.325003 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:49:45.325237 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:49:45.325820 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:49:45.325843 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:49:45.328695 (+ 2852us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:49:45.332084 (+ 3389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:49:45.332096 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:49:45.332104 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:49:45.332581 (+ 477us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:49:45.332990 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:49:45.332994 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:49:45.332998 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:49:45.333191 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:49:45.334024 (+ 833us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:49:45.334033 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:49:45.334224 (+ 191us) client_negotiation.cc:769] Sending connection context
0926 03:49:45.334440 (+ 216us) client_negotiation.cc:240] Negotiation successful
0926 03:49:45.334804 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":583,"thread_start_us":368,"threads_started":1}
I20260926 03:49:45.335417 23775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:49:45.318942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49462 (local address 127.0.194.254:38429)
0926 03:49:45.320390 (+ 1448us) server_negotiation.cc:206] Beginning negotiation
0926 03:49:45.320397 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:49:45.320420 (+ 23us) server_negotiation.cc:407] Connection header received
0926 03:49:45.320484 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:49:45.320489 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:49:45.320592 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:49:45.320717 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:49:45.321780 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:49:45.322674 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:49:45.324295 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:49:45.324756 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:49:45.324893 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:49:45.328888 (+ 3995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:49:45.328893 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:49:45.328897 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:49:45.328937 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:49:45.331959 (+ 3022us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:49:45.332698 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:49:45.332706 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:49:45.332711 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:49:45.332873 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:49:45.333342 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:49:45.333349 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:49:45.333353 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:49:45.333752 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:49:45.333918 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0926 03:49:45.334599 (+ 681us) server_negotiation.cc:299] Negotiation successful
0926 03:49:45.334875 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1305,"thread_start_us":1157,"threads_started":1}
I20260926 03:49:45.337028 23770 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.194.253" port: 43771 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 03:49:45 dist-test-slave-n2th krb5kdc[23541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394585, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.252@KRBTEST.COM for kudu/127.0.194.253@KRBTEST.COM
Sep 26 03:49:45 dist-test-slave-n2th krb5kdc[23541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394583, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.194.254@KRBTEST.COM
I20260926 03:49:45.350889 23774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:49:45.337808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.253:43771 (local address 127.0.0.1:55576)
0926 03:49:45.337959 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0926 03:49:45.337978 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 03:49:45.338072 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:49:45.338378 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:49:45.338381 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:49:45.338682 (+ 301us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:49:45.338883 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:49:45.338891 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:49:45.339878 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:49:45.339883 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:49:45.340978 (+ 1095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:49:45.340992 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:49:45.341699 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:49:45.341705 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:49:45.341880 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:49:45.342408 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:49:45.342423 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:49:45.344590 (+ 2167us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:49:45.348062 (+ 3472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:49:45.348068 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:49:45.348074 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:49:45.348596 (+ 522us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:49:45.349054 (+ 458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:49:45.349057 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:49:45.349059 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:49:45.349251 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:49:45.349887 (+ 636us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:49:45.349892 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:49:45.350051 (+ 159us) client_negotiation.cc:769] Sending connection context
0926 03:49:45.350219 (+ 168us) client_negotiation.cc:240] Negotiation successful
0926 03:49:45.350495 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260926 03:49:45.350982 23700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:49:45.337800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55576 (local address 127.0.194.253:43771)
0926 03:49:45.337995 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0926 03:49:45.338003 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:49:45.338021 (+ 18us) server_negotiation.cc:407] Connection header received
0926 03:49:45.338195 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:49:45.338200 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:49:45.338276 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:49:45.338377 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:49:45.339027 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:49:45.339740 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:49:45.341135 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:49:45.341540 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:49:45.341676 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:49:45.344777 (+ 3101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:49:45.344782 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:49:45.344785 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:49:45.344818 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:49:45.347921 (+ 3103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:49:45.348709 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:49:45.348715 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:49:45.348719 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:49:45.348931 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:49:45.349365 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:49:45.349368 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:49:45.349371 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:49:45.349760 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:49:45.349880 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0926 03:49:45.350335 (+ 455us) server_negotiation.cc:299] Negotiation successful
0926 03:49:45.350567 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 03:49:45.351332 23776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:49:45.334802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49472 (local address 127.0.194.254:38429)
0926 03:49:45.335529 (+ 727us) server_negotiation.cc:206] Beginning negotiation
0926 03:49:45.335537 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:49:45.336258 (+ 721us) server_negotiation.cc:407] Connection header received
0926 03:49:45.336442 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:49:45.336448 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:49:45.336517 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:49:45.336642 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:49:45.337896 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:49:45.339039 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:49:45.340466 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:49:45.340887 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:49:45.340980 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:49:45.345552 (+ 4572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:49:45.345557 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:49:45.345560 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:49:45.345590 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:49:45.348381 (+ 2791us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:49:45.349202 (+ 821us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:49:45.349209 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:49:45.349213 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:49:45.349368 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:49:45.349750 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:49:45.349753 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:49:45.349755 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:49:45.350089 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:49:45.350368 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0926 03:49:45.350694 (+ 326us) server_negotiation.cc:299] Negotiation successful
0926 03:49:45.350947 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":563,"thread_start_us":1149,"threads_started":1}
I20260926 03:49:45.352092 23770 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.194.252" port: 40195 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 03:49:45 dist-test-slave-n2th krb5kdc[23541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394585, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.252@KRBTEST.COM for kudu/127.0.194.252@KRBTEST.COM
Sep 26 03:49:45 dist-test-slave-n2th krb5kdc[23541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394583, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.194.253@KRBTEST.COM
I20260926 03:49:45.365288 23774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:49:45.352825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.252:40195 (local address 127.0.0.1:34816)
0926 03:49:45.353203 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0926 03:49:45.353221 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 03:49:45.353324 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:49:45.353602 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:49:45.353606 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:49:45.353979 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:49:45.354104 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:49:45.354111 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:49:45.355061 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:49:45.355067 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:49:45.355893 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:49:45.355903 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:49:45.356572 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:49:45.356578 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:49:45.356749 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:49:45.357322 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:49:45.357338 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:49:45.359366 (+ 2028us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:49:45.362296 (+ 2930us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:49:45.362301 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:49:45.362305 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:49:45.362869 (+ 564us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:49:45.363323 (+ 454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:49:45.363326 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:49:45.363328 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:49:45.363488 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:49:45.364121 (+ 633us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:49:45.364125 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:49:45.364310 (+ 185us) client_negotiation.cc:769] Sending connection context
0926 03:49:45.364490 (+ 180us) client_negotiation.cc:240] Negotiation successful
0926 03:49:45.364840 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"mutex_wait_us":1}
I20260926 03:49:45.365312 23773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:49:45.353035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34816 (local address 127.0.194.252:40195)
0926 03:49:45.353205 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0926 03:49:45.353211 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:49:45.353266 (+ 55us) server_negotiation.cc:407] Connection header received
0926 03:49:45.353431 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:49:45.353437 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:49:45.353491 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:49:45.353582 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:49:45.354228 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:49:45.354951 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:49:45.356024 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:49:45.356413 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:49:45.356550 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:49:45.359528 (+ 2978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:49:45.359539 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:49:45.359543 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:49:45.359570 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:49:45.362160 (+ 2590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:49:45.363006 (+ 846us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:49:45.363016 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:49:45.363019 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:49:45.363209 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:49:45.363609 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:49:45.363616 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:49:45.363619 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:49:45.363990 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:49:45.364132 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0926 03:49:45.364590 (+ 458us) server_negotiation.cc:299] Negotiation successful
0926 03:49:45.364992 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":170,"server-negotiator.queue_time_us":67}
I20260926 03:49:45.366731 23700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:49:45.353097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55590 (local address 127.0.194.253:43771)
0926 03:49:45.353265 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0926 03:49:45.353273 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:49:45.353290 (+ 17us) server_negotiation.cc:407] Connection header received
0926 03:49:45.353336 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:49:45.353341 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:49:45.353407 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:49:45.353503 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:49:45.354659 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:49:45.355414 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:49:45.356567 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:49:45.356992 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:49:45.357125 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:49:45.361166 (+ 4041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:49:45.361174 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:49:45.361183 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:49:45.361214 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:49:45.364134 (+ 2920us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:49:45.364821 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:49:45.364828 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:49:45.364832 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:49:45.364974 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:49:45.365356 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:49:45.365362 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:49:45.365366 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:49:45.365720 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:49:45.365999 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0926 03:49:45.366311 (+ 312us) server_negotiation.cc:299] Negotiation successful
0926 03:49:45.366472 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 03:49:45.371459 23770 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 945ec0d86aab48f58802ba5ecd6fb0a4: Bootstrap starting.
I20260926 03:49:45.371819 779 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 03:49:45.374935 23770 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 945ec0d86aab48f58802ba5ecd6fb0a4: Neither blocks nor log segments found. Creating new log.
I20260926 03:49:45.375995 23770 log.cc:824] T 00000000000000000000000000000000 P 945ec0d86aab48f58802ba5ecd6fb0a4: Log is configured to *not* fsync() on all Append() calls
I20260926 03:49:45.374126 23596 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "ae8444917e8a49959886c1bfd249dd95"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49472
I20260926 03:49:45.378907 23770 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 945ec0d86aab48f58802ba5ecd6fb0a4: No bootstrap required, opened a new log
I20260926 03:49:45.385936 23770 raft_consensus.cc:348] T 00000000000000000000000000000000 P 945ec0d86aab48f58802ba5ecd6fb0a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae8444917e8a49959886c1bfd249dd95" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 38429 } } peers { permanent_uuid: "35ccc0996edd48bd9c5ff73e1264394e" member_type: VOTER last_known_addr { host: "127.0.194.253" port: 43771 } } peers { permanent_uuid: "945ec0d86aab48f58802ba5ecd6fb0a4" member_type: VOTER last_known_addr { host: "127.0.194.252" port: 40195 } }
I20260926 03:49:45.386293 23770 raft_consensus.cc:729] T 00000000000000000000000000000000 P 945ec0d86aab48f58802ba5ecd6fb0a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 945ec0d86aab48f58802ba5ecd6fb0a4, State: Initialized, Role: FOLLOWER
I20260926 03:49:45.386772 23770 consensus_queue.cc:260] T 00000000000000000000000000000000 P 945ec0d86aab48f58802ba5ecd6fb0a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae8444917e8a49959886c1bfd249dd95" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 38429 } } peers { permanent_uuid: "35ccc0996edd48bd9c5ff73e1264394e" member_type: VOTER last_known_addr { host: "127.0.194.253" port: 43771 } } peers { permanent_uuid: "945ec0d86aab48f58802ba5ecd6fb0a4" member_type: VOTER last_known_addr { host: "127.0.194.252" port: 40195 } }
I20260926 03:49:45.387827 23778 sys_catalog.cc:455] T 00000000000000000000000000000000 P 945ec0d86aab48f58802ba5ecd6fb0a4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae8444917e8a49959886c1bfd249dd95" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 38429 } } peers { permanent_uuid: "35ccc0996edd48bd9c5ff73e1264394e" member_type: VOTER last_known_addr { host: "127.0.194.253" port: 43771 } } peers { permanent_uuid: "945ec0d86aab48f58802ba5ecd6fb0a4" member_type: VOTER last_known_addr { host: "127.0.194.252" port: 40195 } } }
I20260926 03:49:45.388228 23778 sys_catalog.cc:458] T 00000000000000000000000000000000 P 945ec0d86aab48f58802ba5ecd6fb0a4 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 03:49:45.388142 23621 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.194.252" port: 40195 } attempt: 4
I20260926 03:49:45.389040 23770 sys_catalog.cc:564] T 00000000000000000000000000000000 P 945ec0d86aab48f58802ba5ecd6fb0a4 [sys.catalog]: configured and running, proceeding with master startup.
Sep 26 03:49:45 dist-test-slave-n2th krb5kdc[23541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394584, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.254@KRBTEST.COM for kudu/127.0.194.252@KRBTEST.COM
I20260926 03:49:45.405016 23623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:49:45.388829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.252:40195 (local address 127.0.0.1:34830)
0926 03:49:45.389115 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0926 03:49:45.389131 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 03:49:45.389273 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:49:45.389732 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:49:45.389738 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:49:45.390184 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:49:45.390402 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:49:45.390417 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:49:45.391681 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:49:45.391687 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:49:45.392671 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:49:45.392684 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:49:45.393460 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:49:45.393466 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:49:45.393680 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:49:45.394314 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:49:45.394331 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:49:45.397278 (+ 2947us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:49:45.401961 (+ 4683us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:49:45.401966 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:49:45.401971 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:49:45.402537 (+ 566us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:49:45.402984 (+ 447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:49:45.402989 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:49:45.402993 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:49:45.403144 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:49:45.403929 (+ 785us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:49:45.403936 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:49:45.404131 (+ 195us) client_negotiation.cc:769] Sending connection context
0926 03:49:45.404310 (+ 179us) client_negotiation.cc:240] Negotiation successful
0926 03:49:45.404636 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159}
I20260926 03:49:45.406207 23773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:49:45.388955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34830 (local address 127.0.194.252:40195)
0926 03:49:45.389464 (+ 509us) server_negotiation.cc:206] Beginning negotiation
0926 03:49:45.389468 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:49:45.389482 (+ 14us) server_negotiation.cc:407] Connection header received
0926 03:49:45.389525 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:49:45.389529 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:49:45.389589 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:49:45.389696 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:49:45.390566 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:49:45.391523 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:49:45.392823 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:49:45.393300 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:49:45.393434 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:49:45.397516 (+ 4082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:49:45.397528 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:49:45.397533 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:49:45.397583 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:49:45.401824 (+ 4241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:49:45.402664 (+ 840us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:49:45.402672 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:49:45.402677 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:49:45.402867 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:49:45.403268 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:49:45.403279 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:49:45.403285 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:49:45.403770 (+ 485us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:49:45.405285 (+ 1515us) server_negotiation.cc:1035] Waiting for connection context
0926 03:49:45.405437 (+ 152us) server_negotiation.cc:299] Negotiation successful
0926 03:49:45.405707 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":393}
W20260926 03:49:45.406343 23789 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 945ec0d86aab48f58802ba5ecd6fb0a4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 03:49:45.406471 23789 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 03:49:45.403966 23703 master_runner.cc:186] Error getting master registration for 127.0.194.254:38429: OK, instance_id { permanent_uuid: "ae8444917e8a49959886c1bfd249dd95" instance_seqno: 1790394584248149 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 03:49:45.412768 23703 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 03:49:45.417207 23621 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae8444917e8a49959886c1bfd249dd95: Bootstrap starting.
I20260926 03:49:45.421396 23621 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ae8444917e8a49959886c1bfd249dd95: Neither blocks nor log segments found. Creating new log.
I20260926 03:49:45.422710 23621 log.cc:824] T 00000000000000000000000000000000 P ae8444917e8a49959886c1bfd249dd95: Log is configured to *not* fsync() on all Append() calls
I20260926 03:49:45.426294 23621 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae8444917e8a49959886c1bfd249dd95: No bootstrap required, opened a new log
I20260926 03:49:45.434008 23621 raft_consensus.cc:348] T 00000000000000000000000000000000 P ae8444917e8a49959886c1bfd249dd95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae8444917e8a49959886c1bfd249dd95" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 38429 } } peers { permanent_uuid: "35ccc0996edd48bd9c5ff73e1264394e" member_type: VOTER last_known_addr { host: "127.0.194.253" port: 43771 } } peers { permanent_uuid: "945ec0d86aab48f58802ba5ecd6fb0a4" member_type: VOTER last_known_addr { host: "127.0.194.252" port: 40195 } }
I20260926 03:49:45.434459 23621 raft_consensus.cc:729] T 00000000000000000000000000000000 P ae8444917e8a49959886c1bfd249dd95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae8444917e8a49959886c1bfd249dd95, State: Initialized, Role: FOLLOWER
I20260926 03:49:45.435003 23621 consensus_queue.cc:260] T 00000000000000000000000000000000 P ae8444917e8a49959886c1bfd249dd95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae8444917e8a49959886c1bfd249dd95" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 38429 } } peers { permanent_uuid: "35ccc0996edd48bd9c5ff73e1264394e" member_type: VOTER last_known_addr { host: "127.0.194.253" port: 43771 } } peers { permanent_uuid: "945ec0d86aab48f58802ba5ecd6fb0a4" member_type: VOTER last_known_addr { host: "127.0.194.252" port: 40195 } }
I20260926 03:49:45.436002 23792 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae8444917e8a49959886c1bfd249dd95 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae8444917e8a49959886c1bfd249dd95" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 38429 } } peers { permanent_uuid: "35ccc0996edd48bd9c5ff73e1264394e" member_type: VOTER last_known_addr { host: "127.0.194.253" port: 43771 } } peers { permanent_uuid: "945ec0d86aab48f58802ba5ecd6fb0a4" member_type: VOTER last_known_addr { host: "127.0.194.252" port: 40195 } } }
I20260926 03:49:45.436390 23792 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae8444917e8a49959886c1bfd249dd95 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 03:49:45.437377 23621 sys_catalog.cc:564] T 00000000000000000000000000000000 P ae8444917e8a49959886c1bfd249dd95 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:49:45.437445 23596 raft_consensus.cc:482] T 00000000000000000000000000000000 P ae8444917e8a49959886c1bfd249dd95 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260926 03:49:45.437668 23596 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ae8444917e8a49959886c1bfd249dd95 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:49:45.440505 23596 raft_consensus.cc:504] T 00000000000000000000000000000000 P ae8444917e8a49959886c1bfd249dd95 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae8444917e8a49959886c1bfd249dd95" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 38429 } } peers { permanent_uuid: "35ccc0996edd48bd9c5ff73e1264394e" member_type: VOTER last_known_addr { host: "127.0.194.253" port: 43771 } } peers { permanent_uuid: "945ec0d86aab48f58802ba5ecd6fb0a4" member_type: VOTER last_known_addr { host: "127.0.194.252" port: 40195 } }
I20260926 03:49:45.441805 23596 leader_election.cc:288] T 00000000000000000000000000000000 P ae8444917e8a49959886c1bfd249dd95 [CANDIDATE]: Term 1 election: Requested vote from peers 35ccc0996edd48bd9c5ff73e1264394e (127.0.194.253:43771), 945ec0d86aab48f58802ba5ecd6fb0a4 (127.0.194.252:40195)
I20260926 03:49:45.443545 23745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ae8444917e8a49959886c1bfd249dd95" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "945ec0d86aab48f58802ba5ecd6fb0a4"
I20260926 03:49:45.444169 23745 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 945ec0d86aab48f58802ba5ecd6fb0a4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:49:45.450403 23745 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 945ec0d86aab48f58802ba5ecd6fb0a4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae8444917e8a49959886c1bfd249dd95 in term 1.
I20260926 03:49:45.451756 23573 leader_election.cc:302] T 00000000000000000000000000000000 P ae8444917e8a49959886c1bfd249dd95 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 945ec0d86aab48f58802ba5ecd6fb0a4, ae8444917e8a49959886c1bfd249dd95; no voters:
I20260926 03:49:45.452378 23792 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ae8444917e8a49959886c1bfd249dd95 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:49:45.464694 23792 raft_consensus.cc:686] T 00000000000000000000000000000000 P ae8444917e8a49959886c1bfd249dd95 [term 1 LEADER]: Becoming Leader. State: Replica: ae8444917e8a49959886c1bfd249dd95, State: Running, Role: LEADER
I20260926 03:49:45.466274 23792 consensus_queue.cc:237] T 00000000000000000000000000000000 P ae8444917e8a49959886c1bfd249dd95 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae8444917e8a49959886c1bfd249dd95" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 38429 } } peers { permanent_uuid: "35ccc0996edd48bd9c5ff73e1264394e" member_type: VOTER last_known_addr { host: "127.0.194.253" port: 43771 } } peers { permanent_uuid: "945ec0d86aab48f58802ba5ecd6fb0a4" member_type: VOTER last_known_addr { host: "127.0.194.252" port: 40195 } }
I20260926 03:49:45.480288 23798 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae8444917e8a49959886c1bfd249dd95 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ae8444917e8a49959886c1bfd249dd95. Latest consensus state: current_term: 1 leader_uuid: "ae8444917e8a49959886c1bfd249dd95" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae8444917e8a49959886c1bfd249dd95" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 38429 } } peers { permanent_uuid: "35ccc0996edd48bd9c5ff73e1264394e" member_type: VOTER last_known_addr { host: "127.0.194.253" port: 43771 } } peers { permanent_uuid: "945ec0d86aab48f58802ba5ecd6fb0a4" member_type: VOTER last_known_addr { host: "127.0.194.252" port: 40195 } } }
I20260926 03:49:45.481263 23798 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae8444917e8a49959886c1bfd249dd95 [sys.catalog]: This master's current role is: LEADER
W20260926 03:49:45.487051 23575 tablet.cc:2369] T 00000000000000000000000000000000 P ae8444917e8a49959886c1bfd249dd95: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 03:49:45.492051 23807 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:49:45.499433 23776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:49:45.468853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49478 (local address 127.0.194.254:38429)
0926 03:49:45.469171 (+ 318us) server_negotiation.cc:206] Beginning negotiation
0926 03:49:45.469181 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 03:49:45.479622 (+ 10441us) server_negotiation.cc:407] Connection header received
0926 03:49:45.480169 (+ 547us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:49:45.480178 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:49:45.480301 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:49:45.480384 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:49:45.482852 (+ 2468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:49:45.484156 (+ 1304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:49:45.486500 (+ 2344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:49:45.487297 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:49:45.487400 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:49:45.491301 (+ 3901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:49:45.491307 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:49:45.491311 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:49:45.491357 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:49:45.495494 (+ 4137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:49:45.496351 (+ 857us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:49:45.496356 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:49:45.496359 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:49:45.496528 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:49:45.496991 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:49:45.496998 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:49:45.497004 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:49:45.497361 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:49:45.497632 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0926 03:49:45.498727 (+ 1095us) server_negotiation.cc:299] Negotiation successful
0926 03:49:45.499012 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 03:49:45.506847 23695 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.194.252" port: 40195 } attempt: 5
I20260926 03:49:45.527057 23700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:49:45.475071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55600 (local address 127.0.194.253:43771)
0926 03:49:45.475321 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0926 03:49:45.475330 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:49:45.494516 (+ 19186us) server_negotiation.cc:407] Connection header received
0926 03:49:45.495322 (+ 806us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:49:45.495330 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:49:45.495430 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:49:45.495505 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:49:45.504068 (+ 8563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:49:45.505274 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:49:45.509242 (+ 3968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:49:45.510173 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:49:45.510300 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:49:45.517573 (+ 7273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:49:45.517585 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:49:45.517593 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:49:45.517654 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:49:45.521586 (+ 3932us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:49:45.523082 (+ 1496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:49:45.523091 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:49:45.523096 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:49:45.523298 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:49:45.524068 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:49:45.524075 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:49:45.524078 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:49:45.524492 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:49:45.524774 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0926 03:49:45.525716 (+ 942us) server_negotiation.cc:299] Negotiation successful
0926 03:49:45.526206 (+ 490us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
W20260926 03:49:45.534410 23724 tablet.cc:2369] T 00000000000000000000000000000000 P 945ec0d86aab48f58802ba5ecd6fb0a4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 03:49:45.537635 23773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:49:45.501889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34832 (local address 127.0.194.252:40195)
0926 03:49:45.502099 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0926 03:49:45.502108 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:49:45.503160 (+ 1052us) server_negotiation.cc:407] Connection header received
0926 03:49:45.503447 (+ 287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:49:45.503452 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:49:45.503555 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:49:45.503633 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:49:45.505466 (+ 1833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:49:45.506646 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:49:45.518100 (+ 11454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:49:45.519207 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:49:45.519324 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:49:45.528768 (+ 9444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:49:45.528774 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:49:45.528778 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:49:45.528814 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:49:45.533418 (+ 4604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:49:45.534341 (+ 923us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:49:45.534348 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:49:45.534354 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:49:45.534614 (+ 260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:49:45.535120 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:49:45.535126 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:49:45.535130 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:49:45.535794 (+ 664us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:49:45.536308 (+ 514us) server_negotiation.cc:1035] Waiting for connection context
0926 03:49:45.536582 (+ 274us) server_negotiation.cc:299] Negotiation successful
0926 03:49:45.536989 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 03:49:45.546362 23807 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 26 03:49:45 dist-test-slave-n2th krb5kdc[23541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394584, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.194.253@KRBTEST.COM for kudu/127.0.194.252@KRBTEST.COM
I20260926 03:49:45.563832 23699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:49:45.507452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.194.252:40195 (local address 127.0.0.1:34842)
0926 03:49:45.507797 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0926 03:49:45.507817 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 03:49:45.507915 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:49:45.530905 (+ 22990us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:49:45.530912 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:49:45.531411 (+ 499us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:49:45.531619 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:49:45.531630 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:49:45.536365 (+ 4735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:49:45.536371 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:49:45.537653 (+ 1282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:49:45.537667 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:49:45.546675 (+ 9008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:49:45.546683 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:49:45.547008 (+ 325us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:49:45.547746 (+ 738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:49:45.547772 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:49:45.551595 (+ 3823us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:49:45.560682 (+ 9087us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:49:45.560688 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:49:45.560693 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:49:45.561311 (+ 618us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:49:45.561843 (+ 532us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:49:45.561849 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:49:45.561853 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:49:45.562037 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:49:45.562773 (+ 736us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:49:45.562778 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:49:45.562946 (+ 168us) client_negotiation.cc:769] Sending connection context
0926 03:49:45.563092 (+ 146us) client_negotiation.cc:240] Negotiation successful
0926 03:49:45.563430 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223}
I20260926 03:49:45.566682 23810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:49:45.508120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34842 (local address 127.0.194.252:40195)
0926 03:49:45.528265 (+ 20145us) server_negotiation.cc:206] Beginning negotiation
0926 03:49:45.528270 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:49:45.528293 (+ 23us) server_negotiation.cc:407] Connection header received
0926 03:49:45.528353 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:49:45.528358 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:49:45.528444 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:49:45.533651 (+ 5207us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:49:45.533936 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:49:45.536193 (+ 2257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:49:45.537850 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:49:45.542632 (+ 4782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:49:45.551790 (+ 9158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:49:45.552064 (+ 274us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:49:45.552071 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:49:45.552076 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:49:45.552111 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:49:45.556064 (+ 3953us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:49:45.561490 (+ 5426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:49:45.561495 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:49:45.561499 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:49:45.561704 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:49:45.562196 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:49:45.562201 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:49:45.562204 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:49:45.562603 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:49:45.565022 (+ 2419us) server_negotiation.cc:1035] Waiting for connection context
0926 03:49:45.565426 (+ 404us) server_negotiation.cc:299] Negotiation successful
0926 03:49:45.565703 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19977,"thread_start_us":20418,"threads_started":1}
I20260926 03:49:45.577592 23695 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 35ccc0996edd48bd9c5ff73e1264394e: Bootstrap starting.
I20260926 03:49:45.581722 23745 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 945ec0d86aab48f58802ba5ecd6fb0a4 [term 1 FOLLOWER]: Refusing update from remote peer ae8444917e8a49959886c1bfd249dd95: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:49:45.582775 23792 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ae8444917e8a49959886c1bfd249dd95 [LEADER]: Connected to new peer: Peer: permanent_uuid: "945ec0d86aab48f58802ba5ecd6fb0a4" member_type: VOTER last_known_addr { host: "127.0.194.252" port: 40195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:49:45.586131 23695 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 35ccc0996edd48bd9c5ff73e1264394e: Neither blocks nor log segments found. Creating new log.
I20260926 03:49:45.591562 23695 log.cc:824] T 00000000000000000000000000000000 P 35ccc0996edd48bd9c5ff73e1264394e: Log is configured to *not* fsync() on all Append() calls
I20260926 03:49:45.568157 23554 master_runner.cc:186] Error getting master registration for 127.0.194.253:43771: OK, instance_id { permanent_uuid: "35ccc0996edd48bd9c5ff73e1264394e" instance_seqno: 1790394584772179 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 03:49:45.595279 23554 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 03:49:45.600193 23695 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 35ccc0996edd48bd9c5ff73e1264394e: No bootstrap required, opened a new log
I20260926 03:49:45.615520 23695 raft_consensus.cc:348] T 00000000000000000000000000000000 P 35ccc0996edd48bd9c5ff73e1264394e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae8444917e8a49959886c1bfd249dd95" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 38429 } } peers { permanent_uuid: "35ccc0996edd48bd9c5ff73e1264394e" member_type: VOTER last_known_addr { host: "127.0.194.253" port: 43771 } } peers { permanent_uuid: "945ec0d86aab48f58802ba5ecd6fb0a4" member_type: VOTER last_known_addr { host: "127.0.194.252" port: 40195 } }
I20260926 03:49:45.616056 23695 raft_consensus.cc:729] T 00000000000000000000000000000000 P 35ccc0996edd48bd9c5ff73e1264394e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35ccc0996edd48bd9c5ff73e1264394e, State: Initialized, Role: FOLLOWER
I20260926 03:49:45.616729 23695 consensus_queue.cc:260] T 00000000000000000000000000000000 P 35ccc0996edd48bd9c5ff73e1264394e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae8444917e8a49959886c1bfd249dd95" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 38429 } } peers { permanent_uuid: "35ccc0996edd48bd9c5ff73e1264394e" member_type: VOTER last_known_addr { host: "127.0.194.253" port: 43771 } } peers { permanent_uuid: "945ec0d86aab48f58802ba5ecd6fb0a4" member_type: VOTER last_known_addr { host: "127.0.194.252" port: 40195 } }
I20260926 03:49:45.618683 23778 sys_catalog.cc:455] T 00000000000000000000000000000000 P 945ec0d86aab48f58802ba5ecd6fb0a4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ae8444917e8a49959886c1bfd249dd95. Latest consensus state: current_term: 1 leader_uuid: "ae8444917e8a49959886c1bfd249dd95" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae8444917e8a49959886c1bfd249dd95" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 38429 } } peers { permanent_uuid: "35ccc0996edd48bd9c5ff73e1264394e" member_type: VOTER last_known_addr { host: "127.0.194.253" port: 43771 } } peers { permanent_uuid: "945ec0d86aab48f58802ba5ecd6fb0a4" member_type: VOTER last_known_addr { host: "127.0.194.252" port: 40195 } } }
I20260926 03:49:45.619138 23778 sys_catalog.cc:458] T 00000000000000000000000000000000 P 945ec0d86aab48f58802ba5ecd6fb0a4 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 03:49:45.623308 23815 mvcc.cc:205] Tried to move back new op lower bound from 7333456222516113408 to 7333456222083739648. Current Snapshot: MvccSnapshot[applied={T|T < 7333456222516113408}]
I20260926 03:49:45.625495 23792 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae8444917e8a49959886c1bfd249dd95 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ae8444917e8a49959886c1bfd249dd95" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae8444917e8a49959886c1bfd249dd95" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 38429 } } peers { permanent_uuid: "35ccc0996edd48bd9c5ff73e1264394e" member_type: VOTER last_known_addr { host: "127.0.194.253" port: 43771 } } peers { permanent_uuid: "945ec0d86aab48f58802ba5ecd6fb0a4" member_type: VOTER last_known_addr { host: "127.0.194.252" port: 40195 } } }
I20260926 03:49:45.625815 23792 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae8444917e8a49959886c1bfd249dd95 [sys.catalog]: This master's current role is: LEADER
I20260926 03:49:45.625597 23819 sys_catalog.cc:455] T 00000000000000000000000000000000 P 35ccc0996edd48bd9c5ff73e1264394e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae8444917e8a49959886c1bfd249dd95" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 38429 } } peers { permanent_uuid: "35ccc0996edd48bd9c5ff73e1264394e" member_type: VOTER last_known_addr { host: "127.0.194.253" port: 43771 } } peers { permanent_uuid: "945ec0d86aab48f58802ba5ecd6fb0a4" member_type: VOTER last_known_addr { host: "127.0.194.252" port: 40195 } } }
I20260926 03:49:45.626032 23819 sys_catalog.cc:458] T 00000000000000000000000000000000 P 35ccc0996edd48bd9c5ff73e1264394e [sys.catalog]: This master's current role is: FOLLOWER
I20260926 03:49:45.628008 23695 sys_catalog.cc:564] T 00000000000000000000000000000000 P 35ccc0996edd48bd9c5ff73e1264394e [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:49:45.628417 23669 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 35ccc0996edd48bd9c5ff73e1264394e [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:49:45.629859 23807 catalog_manager.cc:1347] Generated new cluster ID: 7615aeff5d7a4d8ea15fdcc23056b759
I20260926 03:49:45.629959 23807 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:49:45.628901 23670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ae8444917e8a49959886c1bfd249dd95" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "35ccc0996edd48bd9c5ff73e1264394e"
I20260926 03:49:45.634866 23669 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 35ccc0996edd48bd9c5ff73e1264394e [term 1 FOLLOWER]: Refusing update from remote peer ae8444917e8a49959886c1bfd249dd95: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:49:45.638473 23670 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 35ccc0996edd48bd9c5ff73e1264394e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae8444917e8a49959886c1bfd249dd95 in term 1.
I20260926 03:49:45.639150 23798 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ae8444917e8a49959886c1bfd249dd95 [LEADER]: Connected to new peer: Peer: permanent_uuid: "35ccc0996edd48bd9c5ff73e1264394e" member_type: VOTER last_known_addr { host: "127.0.194.253" port: 43771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:49:45.649441 23778 sys_catalog.cc:455] T 00000000000000000000000000000000 P 945ec0d86aab48f58802ba5ecd6fb0a4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ae8444917e8a49959886c1bfd249dd95" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae8444917e8a49959886c1bfd249dd95" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 38429 } } peers { permanent_uuid: "35ccc0996edd48bd9c5ff73e1264394e" member_type: VOTER last_known_addr { host: "127.0.194.253" port: 43771 } } peers { permanent_uuid: "945ec0d86aab48f58802ba5ecd6fb0a4" member_type: VOTER last_known_addr { host: "127.0.194.252" port: 40195 } } }
I20260926 03:49:45.649727 23778 sys_catalog.cc:458] T 00000000000000000000000000000000 P 945ec0d86aab48f58802ba5ecd6fb0a4 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 03:49:45.666491 23807 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:49:45.668507 23807 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:49:45.678699 23819 sys_catalog.cc:455] T 00000000000000000000000000000000 P 35ccc0996edd48bd9c5ff73e1264394e [sys.catalog]: SysCatalogTable state changed. Reason: New leader ae8444917e8a49959886c1bfd249dd95. Latest consensus state: current_term: 1 leader_uuid: "ae8444917e8a49959886c1bfd249dd95" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae8444917e8a49959886c1bfd249dd95" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 38429 } } peers { permanent_uuid: "35ccc0996edd48bd9c5ff73e1264394e" member_type: VOTER last_known_addr { host: "127.0.194.253" port: 43771 } } peers { permanent_uuid: "945ec0d86aab48f58802ba5ecd6fb0a4" member_type: VOTER last_known_addr { host: "127.0.194.252" port: 40195 } } }
I20260926 03:49:45.679071 23819 sys_catalog.cc:458] T 00000000000000000000000000000000 P 35ccc0996edd48bd9c5ff73e1264394e [sys.catalog]: This master's current role is: FOLLOWER
I20260926 03:49:45.681330 23792 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae8444917e8a49959886c1bfd249dd95 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ae8444917e8a49959886c1bfd249dd95" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae8444917e8a49959886c1bfd249dd95" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 38429 } } peers { permanent_uuid: "35ccc0996edd48bd9c5ff73e1264394e" member_type: VOTER last_known_addr { host: "127.0.194.253" port: 43771 } } peers { permanent_uuid: "945ec0d86aab48f58802ba5ecd6fb0a4" member_type: VOTER last_known_addr { host: "127.0.194.252" port: 40195 } } }
I20260926 03:49:45.681607 23792 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae8444917e8a49959886c1bfd249dd95 [sys.catalog]: This master's current role is: LEADER
I20260926 03:49:45.691758 23807 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ae8444917e8a49959886c1bfd249dd95: Generated new TSK 0
I20260926 03:49:45.692855 23807 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 03:49:45.681984 23834 sys_catalog.cc:455] T 00000000000000000000000000000000 P 35ccc0996edd48bd9c5ff73e1264394e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ae8444917e8a49959886c1bfd249dd95" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae8444917e8a49959886c1bfd249dd95" member_type: VOTER last_known_addr { host: "127.0.194.254" port: 38429 } } peers { permanent_uuid: "35ccc0996edd48bd9c5ff73e1264394e" member_type: VOTER last_known_addr { host: "127.0.194.253" port: 43771 } } peers { permanent_uuid: "945ec0d86aab48f58802ba5ecd6fb0a4" member_type: VOTER last_known_addr { host: "127.0.194.252" port: 40195 } } }
I20260926 03:49:45.694487 23834 sys_catalog.cc:458] T 00000000000000000000000000000000 P 35ccc0996edd48bd9c5ff73e1264394e [sys.catalog]: This master's current role is: FOLLOWER
I20260926 03:49:45.699919 23836 catalog_manager.cc:1259] Loaded cluster ID: 7615aeff5d7a4d8ea15fdcc23056b759
I20260926 03:49:45.700075 23836 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 35ccc0996edd48bd9c5ff73e1264394e: loading cluster ID for follower catalog manager: success
I20260926 03:49:45.704128 23836 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 35ccc0996edd48bd9c5ff73e1264394e: acquiring CA information for follower catalog manager: success
I20260926 03:49:45.732795 23836 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 35ccc0996edd48bd9c5ff73e1264394e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260926 03:49:45.837023 23776 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:49:45.820845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49488 (local address 127.0.194.254:38429)
0926 03:49:45.821059 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0926 03:49:45.821076 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 03:49:45.834544 (+ 13468us) server_negotiation.cc:407] Connection header received
0926 03:49:45.835144 (+ 600us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:49:45.835151 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:49:45.835246 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:49:45.835324 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:49:45.836698 (+ 1374us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49488: BlockingRecv error: recv got EOF from 127.0.0.1:49488 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260926 03:49:45.838382 23700 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:49:45.829388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55602 (local address 127.0.194.253:43771)
0926 03:49:45.829652 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0926 03:49:45.829659 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:49:45.836457 (+ 6798us) server_negotiation.cc:407] Connection header received
0926 03:49:45.836721 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:49:45.836727 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:49:45.836810 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:49:45.836885 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:49:45.838149 (+ 1264us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55602: BlockingRecv error: recv got EOF from 127.0.0.1:55602 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
W20260926 03:49:45.840454 23810 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:49:45.838289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34846 (local address 127.0.194.252:40195)
0926 03:49:45.838477 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0926 03:49:45.838483 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:49:45.838923 (+ 440us) server_negotiation.cc:407] Connection header received
0926 03:49:45.839168 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:49:45.839175 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:49:45.839272 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:49:45.839345 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:49:45.840233 (+ 888us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34846: BlockingRecv error: recv got EOF from 127.0.0.1:34846 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 03:49:45.889201 23700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:49:45.873871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55616 (local address 127.0.194.253:43771)
0926 03:49:45.874070 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0926 03:49:45.874076 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:49:45.881023 (+ 6947us) server_negotiation.cc:407] Connection header received
0926 03:49:45.881582 (+ 559us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:49:45.881602 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:49:45.881820 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:49:45.881967 (+ 147us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:49:45.882707 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:49:45.883887 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:49:45.886014 (+ 2127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:49:45.886684 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:49:45.886844 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:49:45.887628 (+ 784us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:49:45.888367 (+ 739us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:49:45.888507 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0926 03:49:45.888748 (+ 241us) server_negotiation.cc:299] Negotiation successful
0926 03:49:45.888861 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260926 03:49:45.933015 23810 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:49:45.923363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34856 (local address 127.0.194.252:40195)
0926 03:49:45.923553 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0926 03:49:45.923560 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:49:45.930532 (+ 6972us) server_negotiation.cc:407] Connection header received
0926 03:49:45.931142 (+ 610us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:49:45.931152 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:49:45.931301 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:49:45.931387 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:49:45.932684 (+ 1297us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34856: BlockingRecv error: recv got EOF from 127.0.0.1:34856 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 03:49:45.981392 23700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:49:45.967123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55632 (local address 127.0.194.253:43771)
0926 03:49:45.967374 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0926 03:49:45.967382 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:49:45.974314 (+ 6932us) server_negotiation.cc:407] Connection header received
0926 03:49:45.974909 (+ 595us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:49:45.974917 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:49:45.975024 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:49:45.975091 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:49:45.975771 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:49:45.976943 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:49:45.979086 (+ 2143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:49:45.979800 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:49:45.979899 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:49:45.980524 (+ 625us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:49:45.980687 (+ 163us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:49:45.980748 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 03:49:45.980979 (+ 231us) server_negotiation.cc:299] Negotiation successful
0926 03:49:45.981074 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
W20260926 03:49:46.024952 23810 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:49:46.015623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34862 (local address 127.0.194.252:40195)
0926 03:49:46.015880 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0926 03:49:46.015887 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:49:46.022855 (+ 6968us) server_negotiation.cc:407] Connection header received
0926 03:49:46.023382 (+ 527us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:49:46.023389 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:49:46.023495 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:49:46.023580 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:49:46.024696 (+ 1116us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34862: BlockingRecv error: recv got EOF from 127.0.0.1:34862 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
I20260926 03:49:46.090286 23700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:49:46.064364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55642 (local address 127.0.194.253:43771)
0926 03:49:46.064568 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0926 03:49:46.064574 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:49:46.083463 (+ 18889us) server_negotiation.cc:407] Connection header received
0926 03:49:46.083695 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:49:46.083701 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:49:46.083800 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:49:46.083910 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:49:46.084622 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:49:46.085795 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:49:46.087867 (+ 2072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:49:46.088555 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:49:46.088693 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:49:46.089307 (+ 614us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:49:46.089493 (+ 186us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:49:46.089573 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 03:49:46.089816 (+ 243us) server_negotiation.cc:299] Negotiation successful
0926 03:49:46.089913 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
W20260926 03:49:46.100904 23810 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:49:46.097517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34872 (local address 127.0.194.252:40195)
0926 03:49:46.097697 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0926 03:49:46.097703 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:49:46.099188 (+ 1485us) server_negotiation.cc:407] Connection header received
0926 03:49:46.099461 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:49:46.099467 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:49:46.099568 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:49:46.099687 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:49:46.100631 (+ 944us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34872: BlockingRecv error: recv got EOF from 127.0.0.1:34872 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 03:49:46.115914 23700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:49:46.108489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55656 (local address 127.0.194.253:43771)
0926 03:49:46.108742 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0926 03:49:46.108749 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:49:46.109374 (+ 625us) server_negotiation.cc:407] Connection header received
0926 03:49:46.109613 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:49:46.109619 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:49:46.109716 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:49:46.109830 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:49:46.110340 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:49:46.111541 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:49:46.113620 (+ 2079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:49:46.114266 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:49:46.114403 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:49:46.115069 (+ 666us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:49:46.115222 (+ 153us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:49:46.115336 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0926 03:49:46.115511 (+ 175us) server_negotiation.cc:299] Negotiation successful
0926 03:49:46.115607 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112}
W20260926 03:49:46.127225 23810 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:49:46.124751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34878 (local address 127.0.194.252:40195)
0926 03:49:46.124991 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0926 03:49:46.124999 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:49:46.125574 (+ 575us) server_negotiation.cc:407] Connection header received
0926 03:49:46.125797 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:49:46.125802 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:49:46.125898 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:49:46.126011 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:49:46.126984 (+ 973us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34878: BlockingRecv error: recv got EOF from 127.0.0.1:34878 (error 108)
Metrics: {"server-negotiator.queue_time_us":109}
I20260926 03:49:46.148335 23700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:49:46.140834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55670 (local address 127.0.194.253:43771)
0926 03:49:46.141041 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0926 03:49:46.141048 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:49:46.141636 (+ 588us) server_negotiation.cc:407] Connection header received
0926 03:49:46.141878 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:49:46.141883 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:49:46.141982 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:49:46.142111 (+ 129us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:49:46.142616 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:49:46.143883 (+ 1267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:49:46.145958 (+ 2075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:49:46.146827 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:49:46.146988 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:49:46.147483 (+ 495us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:49:46.147660 (+ 177us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:49:46.147747 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0926 03:49:46.147926 (+ 179us) server_negotiation.cc:299] Negotiation successful
0926 03:49:46.148028 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
W20260926 03:49:46.158650 23810 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:49:46.156300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34884 (local address 127.0.194.252:40195)
0926 03:49:46.156493 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0926 03:49:46.156502 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:49:46.157120 (+ 618us) server_negotiation.cc:407] Connection header received
0926 03:49:46.157318 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:49:46.157325 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:49:46.157421 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:49:46.157526 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:49:46.158418 (+ 892us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34884: BlockingRecv error: recv got EOF from 127.0.0.1:34884 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 03:49:46.184456 23700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:49:46.178381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55672 (local address 127.0.194.253:43771)
0926 03:49:46.178596 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0926 03:49:46.178608 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 03:49:46.179001 (+ 393us) server_negotiation.cc:407] Connection header received
0926 03:49:46.179197 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:49:46.179203 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:49:46.179301 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:49:46.179417 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:49:46.179833 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:49:46.181005 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:49:46.182495 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:49:46.183193 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:49:46.183341 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:49:46.183688 (+ 347us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:49:46.183837 (+ 149us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:49:46.183934 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0926 03:49:46.184027 (+ 93us) server_negotiation.cc:299] Negotiation successful
0926 03:49:46.184124 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
W20260926 03:49:46.193413 23810 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:49:46.191248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34890 (local address 127.0.194.252:40195)
0926 03:49:46.191698 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0926 03:49:46.191703 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:49:46.191791 (+ 88us) server_negotiation.cc:407] Connection header received
0926 03:49:46.191993 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:49:46.191997 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:49:46.192092 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:49:46.192176 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:49:46.193172 (+ 996us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34890: BlockingRecv error: recv got EOF from 127.0.0.1:34890 (error 108)
Metrics: {"server-negotiator.queue_time_us":336}
I20260926 03:49:46.235903 23700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:49:46.229535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55682 (local address 127.0.194.253:43771)
0926 03:49:46.229797 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0926 03:49:46.229803 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:49:46.230177 (+ 374us) server_negotiation.cc:407] Connection header received
0926 03:49:46.230389 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:49:46.230394 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:49:46.230532 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:49:46.230682 (+ 150us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:49:46.231112 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:49:46.232308 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:49:46.233746 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:49:46.234408 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:49:46.234560 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:49:46.235028 (+ 468us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:49:46.235188 (+ 160us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:49:46.235303 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0926 03:49:46.235421 (+ 118us) server_negotiation.cc:299] Negotiation successful
0926 03:49:46.235517 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
W20260926 03:49:46.244796 23810 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:49:46.242642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34904 (local address 127.0.194.252:40195)
0926 03:49:46.242897 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0926 03:49:46.242905 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:49:46.243283 (+ 378us) server_negotiation.cc:407] Connection header received
0926 03:49:46.243489 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:49:46.243493 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:49:46.243565 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:49:46.243652 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:49:46.244570 (+ 918us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34904: BlockingRecv error: recv got EOF from 127.0.0.1:34904 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
I20260926 03:49:46.318570 23700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:49:46.312448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55690 (local address 127.0.194.253:43771)
0926 03:49:46.312692 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0926 03:49:46.312698 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:49:46.313067 (+ 369us) server_negotiation.cc:407] Connection header received
0926 03:49:46.313267 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:49:46.313271 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:49:46.313344 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:49:46.313426 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:49:46.313929 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:49:46.314716 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:49:46.316783 (+ 2067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:49:46.317223 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:49:46.317320 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:49:46.317891 (+ 571us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:49:46.318001 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:49:46.318070 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 03:49:46.318219 (+ 149us) server_negotiation.cc:299] Negotiation successful
0926 03:49:46.318287 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
W20260926 03:49:46.326589 23810 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:49:46.324630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34912 (local address 127.0.194.252:40195)
0926 03:49:46.324861 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0926 03:49:46.324869 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:49:46.325195 (+ 326us) server_negotiation.cc:407] Connection header received
0926 03:49:46.325411 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:49:46.325417 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:49:46.325496 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:49:46.325613 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:49:46.326374 (+ 761us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34912: BlockingRecv error: recv got EOF from 127.0.0.1:34912 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 03:49:46.408517 23789 catalog_manager.cc:1259] Loaded cluster ID: 7615aeff5d7a4d8ea15fdcc23056b759
I20260926 03:49:46.408666 23789 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 945ec0d86aab48f58802ba5ecd6fb0a4: loading cluster ID for follower catalog manager: success
I20260926 03:49:46.411219 23789 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 945ec0d86aab48f58802ba5ecd6fb0a4: acquiring CA information for follower catalog manager: success
I20260926 03:49:46.413053 23789 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 945ec0d86aab48f58802ba5ecd6fb0a4: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 03:49:46.465181 23700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:49:46.460434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55694 (local address 127.0.194.253:43771)
0926 03:49:46.460609 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0926 03:49:46.460613 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:49:46.460943 (+ 330us) server_negotiation.cc:407] Connection header received
0926 03:49:46.461108 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:49:46.461113 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:49:46.461183 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:49:46.461273 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:49:46.461654 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:49:46.462312 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:49:46.463712 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:49:46.464119 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:49:46.464231 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:49:46.464592 (+ 361us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:49:46.464703 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:49:46.464771 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 03:49:46.464896 (+ 125us) server_negotiation.cc:299] Negotiation successful
0926 03:49:46.464961 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 03:49:46.476693 23810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:49:46.470911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34922 (local address 127.0.194.252:40195)
0926 03:49:46.471132 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0926 03:49:46.471138 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:49:46.471418 (+ 280us) server_negotiation.cc:407] Connection header received
0926 03:49:46.471655 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:49:46.471659 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:49:46.471807 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:49:46.471930 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:49:46.472353 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:49:46.473059 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:49:46.474350 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:49:46.474863 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:49:46.475003 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:49:46.475497 (+ 494us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:49:46.476092 (+ 595us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:49:46.476260 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0926 03:49:46.476319 (+ 59us) server_negotiation.cc:299] Negotiation successful
0926 03:49:46.476402 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 03:49:46.490761 23700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:49:46.482725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55696 (local address 127.0.194.253:43771)
0926 03:49:46.482946 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0926 03:49:46.482955 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:49:46.483881 (+ 926us) server_negotiation.cc:407] Connection header received
0926 03:49:46.484385 (+ 504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:49:46.484392 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:49:46.484482 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:49:46.484571 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:49:46.485286 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:49:46.486431 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:49:46.488386 (+ 1955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:49:46.488939 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:49:46.489079 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:49:46.489938 (+ 859us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:49:46.490083 (+ 145us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:49:46.490176 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0926 03:49:46.490349 (+ 173us) server_negotiation.cc:299] Negotiation successful
0926 03:49:46.490437 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 03:49:46.491345 23941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:49:46.482536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49498 (local address 127.0.194.254:38429)
0926 03:49:46.484369 (+ 1833us) server_negotiation.cc:206] Beginning negotiation
0926 03:49:46.484375 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:49:46.484392 (+ 17us) server_negotiation.cc:407] Connection header received
0926 03:49:46.484505 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:49:46.484513 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:49:46.484747 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:49:46.484935 (+ 188us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:49:46.485576 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:49:46.486370 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:49:46.488356 (+ 1986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:49:46.489030 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:49:46.489157 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:49:46.489833 (+ 676us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:49:46.490371 (+ 538us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:49:46.490906 (+ 535us) server_negotiation.cc:1035] Waiting for connection context
0926 03:49:46.490966 (+ 60us) server_negotiation.cc:299] Negotiation successful
0926 03:49:46.491042 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1722,"thread_start_us":1628,"threads_started":1}
I20260926 03:49:46.491984 23810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:49:46.484230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34938 (local address 127.0.194.252:40195)
0926 03:49:46.484423 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0926 03:49:46.484431 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:49:46.485027 (+ 596us) server_negotiation.cc:407] Connection header received
0926 03:49:46.485392 (+ 365us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:49:46.485400 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:49:46.485485 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:49:46.485575 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:49:46.486312 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:49:46.487483 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:49:46.489649 (+ 2166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:49:46.490260 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:49:46.490394 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:49:46.491221 (+ 827us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:49:46.491353 (+ 132us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:49:46.491403 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0926 03:49:46.491633 (+ 230us) server_negotiation.cc:299] Negotiation successful
0926 03:49:46.491709 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 03:49:46.495903 779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu pid 23554
I20260926 03:49:46.497144 23616 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:49:46.610373 779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu with pid 23554
W20260926 03:49:46.631837 23721 connection.cc:535] server connection from 127.0.0.1:34830 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 03:49:46.632653 779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu pid 23628
I20260926 03:49:46.633876 23690 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:49:46.747747 779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu with pid 23628
I20260926 03:49:46.768749 779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu pid 23703
I20260926 03:49:46.769968 23765 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:49:46.877897 779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktGRWtf/build/asan/bin/kudu with pid 23703
2026-09-26T03:49:46Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (4508 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (9474 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (365289 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST